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Abstract. The infinite-dimensional unitary group U(oo) is the inductive limit of 
growing compact unitary groups U (N) . In this paper we solve a problem of har- 
monic analysis on J7(oo) stated in [013]. The problem consists in computing spectral 
decomposition for a remarkable 4-parameter family of characters of U(oo). These 
characters generate representations which should be viewed as analogs of nonexisting 
regular representation of U(po). 

The spectral decomposition of a character of U(oo) is described by the spectral 
measure which lives on an infinite-dimensional space Q of indecomposable characters. 
The key idea which allows us to solve the problem is to embed into the space 
of point configurations on the real line without 2 points. This turns the spectral 
measure into a stochastic point process on the real line. The main result of the paper 
is a complete description of the processes corresponding to our concrete family of 
characters. We prove that each of the processes is a determinantal point process. 
That is, its correlation functions have determinantal form with a certain kernel. 
Our kernels have a special 'integrable' form and are expressed through the Gauss 
hypergeometric function. 

From the analytic point of view, the problem of computing the correlation kernels 
can be reduced to a problem of evaluating uniform asymptotics of certain discrete 
orthogonal polynomials studied earlier by Richard Askey and Peter Lesky. One dif- 
ficulty lies in the fact that we need to compute the asymptotics in the oscillatory 
regime with the period of oscillations tending to 0. We do this by expressing the 
polynomials in terms of a solution of a discrete Riemann-Hilbert problem and com- 
puting the (non-oscillatory) asymptotics of this solution. 

From the point of view of statistical physics, we study thermodynamic limit of 
a discrete log-gas system. An interesting feature of this log— gas is that its density 
function is asymptotically equal to the characteristic function of an interval. Our 
point processes describe how different the random particle configuration is from the 
typical 'densely packed' configuration. 

In simpler situations of harmonic analysis on infinite symmetric group and har- 
monic analysis of unitarily invariant measures on infinite hermitian matrices similar 
results were obtained in our papers [BOl], [B02], [B04]. 
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Introduction 

(a) Preface. We tried to make this work accessible and interesting for a wide 
category of readers. So we start with a brief explanation of the concepts that enter 
in the title. 

The purpose of harmonic analysis is to decompose natural representations of a 
given group on irreducible representations. By natural representations we mean 
those representations that are produced, in a natural way, from the group itself. 
For instance, this can be the regular representation, which is realized in the L 2 
space on the group, or a quasiregular representation, which is built from the action 
of the group on a homogeneous space. 

In practice, a natural representation often comes together with a distinguished 
cyclic vector. Then the decomposition into irreducibles is governed by a measure, 
which may be called the spectral measure. The spectral measure lives on the dual 
space to the group, the points of the dual being the irreducible unitary repre- 
sentations. There is a useful analogy in analysis: expanding a given function on 
eigenfunctions of a self-adjoint operator. Here the spectrum of the operator is a 
counterpart of the dual space. 

If our distinguished vector lies in the Hilbert space of the representation, then 
the spectral measure has finite mass and can be made a probability measure. 1 

Now let us turn to point processes (or random point fields), which form a special 
class of stochastic processes. In general, a stochastic process is a discrete or con- 
tinual family of random variables, while a point process (or random point field) is 
a random point configuration. By a (nonrandom) point configuration we mean an 
unordered collection of points in a locally compact space X. This collection may be 
finite or countably infinite, but it cannot have accumulation points in X. To define 
a point process on X, we have to specify a probability measure on Conf(X), the set 
of all point configurations. 

The classical example is the Poisson process, which is employed in a lot of prob- 
abilistic models and constructions. Another important example (or rather a class 
of examples) comes from random matrix theory. Given a probability measure on a 
space of N x N matrices, we pass to the matrix eigenvalues and get in this way a 
random A-point configuration. In a suitable scaling limit transition (as A — > oo), 
it turns into a point process living on infinite point configurations. 

As long as we are dealing with 'conventional' groups (finite groups, compact 
groups, real or p-adic reductive groups, etc.), representation theory seems to have 

1 It may well happen that the distinguished vector belongs to an extension of the Hilbert space 
(just as in analysis, one may well be interested in expanding a function which is not square 
integrable). For instance, in the case of the regular representation of a Lie group one usually takes 
the delta function at the unity of the group, which is not an element of L 2 . In such a situation 
the spectral measure is infinite. However, we shall deal with finite spectral measures only. 
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nothing in common with point processes. However, the situation drastically changes 
when we turn to 'big' groups whose irreducible representations depend on infinitely 
many parameters. Two basic examples are the infinite symmetric group S(oo) and 
the infinite-dimensional unitary group U(oo), which are defined as the unions of 
the ascending chains of finite or compact groups 

S(l) c S(2) c S(3) C . . . , U(l) c U(2) C U(3) c . . . , 

respectively. It turns out that for such groups, the clue to the problem of harmonic 
analysis can be found in the theory of point processes. 

The idea is to convert any infinite collection of parameters, which corresponds to 
an irreducible representation, to a point configuration. Then the spectral measure 
defines a point process, and one may try to describe this process (hence the initial 
measure) using appropriate probabilistic tools. 

In [Bl], [B2], [BOl], [P.I-V] we applied this approach to the group S(oo). In the 
present paper we study the more complicated group U(oo). Notice that the point 
processes arising from the spectral measures do not resemble the Poisson process 
but are close to the processes of the random matrix theory. 

Now we proceed to a detailed description of the content of the paper. 

(b) From harmonic analysis on i7(oo) to a random matrix type asymp- 
totic problem. Here we summarize the necessary preliminary results established 
in [013]. For a more detailed review see §§1-3 below. 

The conventional definition of the regular representation is not applicable to 
the group U(oo): one cannot define the L 2 space on this group, because U(oo) is 
not locally compact and hence does not possess an invariant measure. To surpass 
this difficulty we embed U(oo) into a larger space il, which can be defined as a 
projective limit of the spaces U(N) as N — > oo. The space il is no longer a group 
but is still a U(oo) x {7(oo)-space. That is, the two-sided action of U(oo) on itself 
can be extended to an action on the space il. In contrast to U(oo), the space il 
possesses a biinvariant finite measure, which should be viewed as a substitute of 
the nonexisting Haar measure. Moreover, this biinvariant measure is included into 
a whole family {/r s )} s£ c of measures with good transformation properties. 2 Using 
the measures we explicitly construct a family {T zw } ZjW ^c of representations, 
which seem to be a good substitute of the nonexisting regular representation. 3 In 
our understanding, the T zw 's are 'natural representations', and we state the problem 
of harmonic analysis on U(oo) as follows: 

Problem 1. Decompose the representations T zw on irreducible representations. 
This initial formulation then undergoes a few changes. 

2 The idea to enlarge an infinite-dimensional space in order to build measures with good trans- 
formation properties is well known. This is a standard device in measure theory on linear spaces, 
but there are not so much works where it is applied to 'curved' spaces (see, however, [Pil], [Ner]). 
For the history of the measures /i( s ) we refer to [013] and [B04]. A parallel construction for the 
symmetric group case is given in [KOV]. 

3 More precisely, the T zw 's are representations of the group {/(oo) x {/(oo). Thus, they are 
a substitute of the biregular representation. The reason why we are dealing with the group 
{/(oo) x {/(oo) and not [/(oo) is explained in [Oil], [012]. We also give in [013] an alternative 
construction of the representations T zw . 
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The first step follows a very general principle of representation theory: reduce the 
spectral decomposition of representations to the decomposition on extreme points 
in a convex set X consisting of certain positive definite functions on the group. 

In our concrete situation, the elements of the set X are positive definite functions 
on U(oo), constant on conjugacy classes and taking the value 1 at the unity. These 
functions are called characters of U (oo). The extreme points of X , or extreme char- 
acters, are known. They are in a one-to-one correspondence, x^ u i with the 
points 10 of an infinite-dimensional region O (the set O and the extreme characters 
are described in §1 below). An arbitrary character x ^ X can be written in 
the form 

X = f X (u;) P(doo), 
Jo. 

where P is a probability measure on O. The measure P is defined uniquely, it is 
called the spectral measure of the character x- 

Now let us return to the representations T zw . We focus on the case when the 
parameters z,w satisfy the condition $t(z + w) > — \. Under this restriction, 
our construction provides a distinguished vector in T zw . The matrix coefficient 
corresponding to this vector can be viewed as a character Xzw of the group U(oo). 
The spectral measure of Xzw is also the spectral measure of the representation T zw 
provided that z and w are not integral. 4 

Furthermore, we remark that the explicit expression of Xzw, viewed as a function 
in four parameters z, z' = z, w, w' = w, correctly defines a character Xz,z',w,w' for 
a wider set £> a dm C C 4 of 'admissible' quadruples (z, z' ,w,w'). The set £> a dm is 
defined by the inequality $l(z + z' + w + w') > — 1 and some extra restrictions, 
see Definition 3.4 below. Actually, the 'admissible' quadruples depend on 4 real 
parameters. 

This leads us to the following reformulation of Problem 1: 

Problem 2. For any (z, z' , w, w') G D a( i m , compute the spectral measure of the 
character Xz,z',w,w'- 

To proceed further we need to explain in what form we express the characters. 
Rather than write them directly as functions on the group U(oo) we prefer to work 
with their 'Fourier coefficients'. Let us explain what this means. 

Recall that the irreducible representations of the compact group U(N) are la- 
beled by the dominant highest weights, which are nothing but Af-tuples of non- 
increasing integers A = (Ai > ••• > Xn)- F° r the reasons which are explained 
in the text we denote the set of all these A's by GTn (here 'GT' is the abbrevi- 
ation of 'Gelfand-Tsetlin'). For each A G GTn we denote by % A the normalized 
character of the irreducible representation with highest weight A. Here the term 
'character' has the conventional meaning, and normalization means division by the 
degree, so that x A (l) = 1- Given a character x G Af, we restrict it to the subgroup 
U(N) C U(oo). Then we get a positive definite function on U(N), constant on 
conjugacy classes and normalized at 1 G U(N). Hence it can be expanded on the 
functions x A , where the coefficients (these are the 'Fourier coefficients' in question) 



4 If z or w is integral then the distinguished vector is not cyclic, and the spectral measure ofx z 
governs the decomposition of a proper subrepresentation of T zw . 

4 



are nonnegative numbers whose sum equals 1: 
x\u(N)= P nWx X ; Pv(A)>0, p Jv(A) = l; N = l,2,.... 

AeGTjv AeGTjv 

Thus, x produces, for any N = 1, 2, . . . , a probability measure Pn on the discrete 
set GTjv- This fact plays an important role in what follows. 

For any character x = Xz,z',w,w' we dispose of an exact expression for the 'Fourier 
coefficients' Pn(X) = Pn(X | z, z' ,w,w'): 

Pn(X | z, z', to, w') = (normalization constant) • | | (A^ — Xj — i + j) 2 

l<i<j<N 

N 1 
x TT t 

Al F(z -Xt + i)T(z' - X l + i)T(w + N + 1 + K- i)T(w' + N + 1 + X t - i) ' 

(0.1) 

Hence we explicitly know the corresponding measures Pn = Pn( ■ \ z, z' , w, w') on 
the sets GTn- Formula (0.1) is the starting point of the present paper. 

In [013] we prove that for any character x G A^, its spectral measure P can be 
obtained as a limit of the measures Pn as N — > oo. More precisely, we define 
embeddings GUV and we show that the pushforwards of the Pat's weakly 

converge to P. 5 

By virtue of this general result, Problem 2 is now reduced to the following 

Problem 3. For any 'admissible' quadruple of parameters (z, z', w, «/), compute the 
limit of the measures Pn{ • \ z, z', w, w') : given by formula (0.1), as iV — > oo. 

This is exactly the problem we are dealing with in the present paper. There is a 
remarkable analogy between Problem 3 and asymptotic problems of random matrix 
theory. We think this fact is important, so that we discuss it below in detail. From 
now on the reader may forget about the initial representation-theoretic motivation: 
we switch to another language. 

(c) Random matrix ensembles, log— gas systems, and determinantal pro- 
cesses. 

Assume we are given a sequence of measures /zi, /Z2, • • • ont and a parameter (3 > 0. 
For any iV = 1,2,..., we introduce a probability distribution Pn on the space of 
ordered iV-tuples of real numbers {x\ > ■ ■ ■ > xn} by 

(N \ N 

Y\[xi,Xi + dxi] J = (normalization constant) • j | \xi — Xj\@ • j^J fj,N(dxi). 
i=l J l<i<j<N i=l 

(0.2) 

Important examples of such distributions come from random matrix ensembles 
(En,I~in), where En is a vector space of matrices (say, of order N) and /jn is a 
probability measure on En- Then x\, . . . , xn are interpreted as the eigenvalues of 
an N x N matrix, and the distribution Pn is induced by the measure ^n- As for 
the parameter it takes values 1, 2,4, depending on the base field. 

For instance, in the Gaussian ensemble, En is the space of real symmetric, 
complex Hermitian or quaternion Hermitian matrices of order N, and fx n is a 



5 The definition of the embeddings GTn ^ ^ is given in §2(c) below. 
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Gaussian measure invariant under the action of the compact group O(N), U(N) or 
Sp(N), respectively. Then (3 = 1,2,4, respectively. 

If is absolutely continuous with respect to the Lebesgue measure then the 
distribution (0.2) is also absolutely continuous, and its density can be written in 
the form 

f t 

Fn(xi, . . . , xn) = (constant) -exp < — (3 I \og\xi — xj\ 1 + Vzv(a^) 

[ Y<i<j<N i=l 

(0-3) 

This can interpreted as the Gibbs measure of a system of N repelling particles 
interacting through a logarithmic Coulomb potential and confined by an external 
potential Vjy In mathematical physics literature such a system is called a log-gas 
system, see, e.g., [Dy]. 

Given a distribution of form (0.2) or (0.3), one is interested in the statistical 
properties of the random configuration x = (xi) as N goes to infinity. A typical 
question concerns the asymptotic behavior of the correlation functions. The n- 
pariicle correlation function, pn(yi, • • • , Un), can be defined as the density of the 
probability to find a 'particle' of the random configuration in each of n infinitesimal 
intervals [y u y t + dyi}. 6 

One can believe that under a suitable limit transition the iV-particle system 
'converges' to a point process — a probability distribution on infinite configura- 
tions of particles. The limit distribution cannot be given by a formula of type (0.2) 
or (0.3). However, it can be characterized by its correlation functions, which pre- 
sumably are limits of the functions pn as N — > oo. The limit transition is usually 
accompanied by a scaling (a change of variables depending on N), and the final 
result may depend on the scaling. See, e.g., [TW]. 

The special case (3 = 2 offers much more possibilities for analysis than the general 
one. This is due to the fact that for (3 = 2, the correlation functions before the limit 
transition are readily expressed through the orthogonal polynomials po,pi, ■ ■ ■ with 
weight /in- Namely, let S^ N \y' ,y") denote the iVth Christoffel-Darboux kernel, 



i=0 



i + ^ PN(y')PN-i(y") -PN-i(y')PN(y") , ,, m 

= (a constant) • ; , y , y G K, 

y'-y" 

and assume (for the sake of simplicity only) that un has a density /at(x). Then 
the correlation functions are given by a simple determinantal formula 



P^ T) (yi,...,y n ) =det 



SWivuvAy/fNivMNiVj) 



71=1,2, 

l<i,j<n 



If the kernel S^ N \y', y")yj fN(y')fN(y") has a limit K(x',x") under a scaling 
limit transition then the limit correlation functions also have a determinantal form, 

p n (xi,...,x n ) =det[K(x i ,Xj)] 1 < i j< n , n = l,2.... (0.4) 



"This is an intuitive definition only. In a rigorous approach one defines the correlation mea- 
sures, see, e.g. [Len, DVJ] and also the beginning of §4 below. 
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The limit kernel can be evaluated if one disposes of an appropriate information 
about the asymptotic properties of the orthogonal polynomials. 

A point process whose correlation functions have the form (0.4) is called deter- 
minantal, and the corresponding kernel K is called the correlation kernel. Finite 
log-gas systems and their scaling limits are examples of determinantal point pro- 
cesses. In these examples, the correlation kernel is symmetric, but this property is 
not necessary. Our study leads to processes with nonsymmetric correlation kernels 
(see (k) below). A comprehensive survey on determinantal point processes is given 
in [So]. 

(d) Lattice log— gas system defined by (0.1). Remark that the expression 
(0.1) can be transformed to the form (0.2). Indeed, given A G G1V, set / = A + p, 
where 

_ ( N-l N-3 JV-3 N-l \ 

p ^ 2 , 2 ,..., 2 , 2 ) 

is the half-sum of positive roots for GL(N). That is, 

h = \i + ^-i, i = l,...,N. 
Then C = {h, . . . , In} is an A^-tuple of distinct numbers belonging to the lattice 

xiN) = r z, n odd, 

\ Z + |, N even. 
The measure (0.1) on A's induces a probability measure on £'s such that 

N 

(Probability of C) = (a constant) • J] (/< - lj) 2 ■ JJ f N (h), (0.5) 

l<i<j<N i=l 

where, for any x E 3i( N > , 

fN{x) 



r(z-x + ^)r(z'-x + ^)r(w + x + ^)r(w' + x + ^) ' 

(0.6) 

Now we see that (0.5) may be viewed as a discrete log-gas system living on the 
lattice 

(e) Askey— Lesky orthogonal polynomials. The orthogonal polynomials that 
are defined by the weight function (0.6) on are rather interesting. To our 

knowledge, they appeared for the first time in Askey's paper [As]. Then they were 
examined in Lesky's papers [Lesl], [Les2]. We propose to call them the Askey-Lesky 
polynomials. More precisely, we reserve this term for the orthogonal polynomials 
that are defined by a weight function on Z of the form 

(0.7) 



T(A - x)T(B - x)T(C + x)T(D + x) ' 

where A, B, C, D are any complex parameters such that (0.7) is nonnegative on Z. 

The Askey-Lesky polynomials are orthogonal polynomials of hypergeometric 
type in the sense of [NSU]. That is, they are eigenf unctions of a difference analog 
of the hypergeometric differential operator. 
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In contrast to classical orthogonal polynomials, the Askey-Lesky polynomials 
form a finite system. This is caused by the fact that (for nonintegral parameters 
A, B, C, D) the weight function has slow decay as x goes to ±00, so that only finitely 
many moments exist. 

The Askey-Lesky polynomials admit an explicit expression in terms of the gener- 
alized hypergeometric series 3-F 2 (a, 6, c; e, /; 1) with unit argument: the parameters 
A, £?, C, D are inserted, in a certain way, in the indices a, 6, c, e, / of the series. This 
allows us to explicitly express the Christoffel-Darboux kernel in terms of the 3-F 2 (l) 
series. 

(f) The two— component gas system. We have just explained how to reduce 
(0.1) to a lattice log-gas system (0.5), for which we are able to evaluate the cor- 
relation functions. To solve Problem 3, we must then pass to the large A limit. 
However, the limit transition that we need here is qualitatively different from typ- 
ical scaling limits of Random Matrix Theory. It can be shown that, as A gets 
large, almost all A particles occupy positions inside (— ^, Note that there are 
exactly A lattice points in this interval, hence, almost all of them are occupied 
by particles. More precisely, for any e > 0, as A — > 00, the number of particles 
outside {—{\ +e)N, (\ + e)A) remains finite almost surely. In other words, this 
means that the density function of our discrete log-gas is asymptotically equal to 
the characteristic function of the A-point set of lattice points inside (— ^r, ^r). 

At first glance, this picture looks discouraging. Indeed, we know that in the 
limit all the particles are densely packed inside (— ^, -y), and there seem to exist 
no nontrivial limit point process. However, the representation theoretic origin of 
the problem leads to the following modification of the model which possesses a 
meaningful scaling limit. 

Let us divide the lattice %( N ) into two parts, which will be denoted by and 

-*-out • 

■v-(N) _ r jy-i N-3 N-3 N-l ] 

x in \ 2 ' 2 > " " " ' 2' 2J' 

_ < jV+3 N+l 1 1 1 f JV+1 N+3 1 

-^out \ • • • > 2 ' 2 J l 2 ' 2 »•••/• 

Here xj^, the 'inner' part, consists of A points of the lattice that lie on the interval 

(-■ |,y), while X^ut , the 'outer' part, is its complement in %( N \ consisting of the 
points outside this interval. . 

Given a configuration £ of A particles sitting at points l\, . . . ,In of the lattice 
%( N \ we assign to it another configuration, A, formed by the particles in 3£ ^ t 
and the holes (i.e., the unoccupied positions) in . Note that A is a finite 
configuration, too. Since the 'interior' part consists of exactly A points, we see 
that in A, there are equally many particles and holes. However, there number is no 
longer fixed, it varies between and 2 A, depending on the mutual location of C and 
SCjn • For instance, if these two sets coincide then X is the empty configuration, 
and if they do not intersect then |A| = 2 A. 

Under the correspondence C 1— > X our random A-particle system turns into a 
random system of particles and holes. Note that £ 1— > A is reversible, so that both 
systems are equivalent. 

Rewriting (0.5) in terms of the configurations X one sees that the new system 
can be viewed as a discrete two-component log-gas system consisting of oppositely 



signed charges. Systems of such a type were earlier investigated in the mathemat- 
ical physics literature (see [AF], [CJ1], [CJ2], [G], [Fl-3] and references therein). 
However, the known concrete models are quite different from our system. 

From what was said above it follows that all but finitely many particles of the 
new system concentrate, for large AT, near the points ±y- This suggests that if we 
shrink our phase space £( N ~) by the factor of N (so that the points ±^ turn into 
±|) then our two-component log-gas should have a well-defined scaling limit. We 
prove that such a limit exists and it constitutes a point process on R \ {±^} which 
we will denote by V. 

As a matter of fact, the process V can be defined directly from the spectral 
measure P of the character Xz,z',w,w' as we explain in §9. Moreover, knowing V is 
almost equivalent to knowing P, see the discussion before Proposition 9.7. Thus, 
we may restate Problem 3 as 

Problem 4- Describe the point process V. 

It turns out that the most convenient way to describe this point process is to 
compute its correlation functions. Since the correlation functions of V define V 
uniquely, we will be solving 

Problem 4' ■ Find the correlation functions of V. 

(g) Two correlation kernels of the two— component log— gas. There are two 
ways of computing the correlation functions of the two-component log-gas sys- 
tem introduced above. The first one is based on the complementation principle, 
see [BOO, Appendix] and §5(c) below, which says that if we have a determinantal 
point process defined on a discrete set 2) = 2)i LI 2)2 then a new process whose point 
configurations consist of particles in 2)i and holes in 2J 2 , is also determinantal. Fur- 
thermore, the correlation kernel of this new process is easily expressed through the 
correlation kernel of the original process. Thus, one way to obtain the correlation 
functions for the two-component log-gas is to apply the complementation principle 
to the (one-component) log-gas (0.1), whose correlation kernel is, essentially, the 
Christoffel-Darboux kernel for Askey-Lesky orthogonal polynomials. Let us denote 
by K^^ pl the correlation kernel for the two-component log-gas obtained in this 
way. 

Another way to compute the correlation functions of out two-component log- 
gas is to notice that this system belongs to the class of point processes with the 
following property: 

The probability of a given point configuration X = {x±, . . . , x n } is given by 

Prob{X} = const • det[L (A °(^, Xj)\l j=1 

where is a x £( N ) matrix, see §6. A simple general theorem shows that 
any point process with this property is determinantal, and its correlation kernel 
KW is given by the relation = L^ N \l + L^)" 1 . 

Thus, we end up with two correlation kernel if'^j and KW of the same point 
process. Of course, these two kernels must not coincide. For example, they may be 
related by conjugation: 

*^(».y) = |§§* ( ' (*.i») 



where </>(•) is an arbitrary nonvanishing function on X^ N \ (The determinants of 
the form det[K(xi, Xj)] for two conjugate kernels are always equal.) We show that 
this is indeed the case, and that the function <fi takes values ±1. Moreover, we 
prove this statement in a more general setting of a two-component log-gas system 
obtained in a similar way by particles-holes exchange from an arbitrary (3 = 2 
discrete log-gas system on the real line. 

(h) Asymptotics. In our concrete situation the function <fi is identically equal to 
1 on the set X^2, an d is equal to (— l) x ~ on the set x[^ . This means that if we 
want to compute the scaling limit of the correlation functions of our two-component 
log-gas system as N — > oo, then only one of the kernels K^ pl and may be 
used for this purpose, because the function does not have a scaling limit. It is 
not hard to guess which kernel is 'the right one' from the asymptotic point of view. 

Indeed, it is easy to verify that the kernel mentioned above has a well- 

defined scaling limit which we will denote by L. It is a (smooth) kernel on R\{±|}. 
It is then quite natural to assume that the kernel = L^(l + L^)" 1 also 

has a scaling limit K such that K = L(l + L) _1 . Although this argument is 
only partially correct (the kernel L does not always define a bounded operator in 
L 2 (M)), it provides good intuition. We prove that for all admissible values of the 
parameters z, z' , w, w', the kernel K^ N ' has a scaling limit K, and this limit kernel 
is the correlation kernel for the point process V. 

Explicit computation of the kernel K is our main result, and we state it in §10. 

(i) Overcoming technical difficulties: Riemann— Hilbert approach. The 

task of computing the scaling limit of K^ N > as N — > oo is by no means easy. As was 
explained above, this kernel coincides, up to a sign, with K^^ pl which, in turn, 
is easily expressible through the Christoffel-Darboux kernel for the Askey-Lesky 
orthogonal polynomials. Thus, Problem 4 (or 4') may be restated as 

Problem 5. Compute the asymptotics of the Askey-Lesky orthogonal polynomials 

Since it is known how to express these polynomials through the 3F2 hyperge- 
ometric series, one might expect that the remaining part is rather smooth and is 
similar to the situation arising in most (3 = 2 random matrix models. That is, in 
the chosen scaling the polynomials converge with all the derivatives to nice analytic 
functions (like sine or Airy) which then enter in the formula for the limit kernel. 

As a matter of fact, this is indeed how things look like on X^ . The limit kernel 
K is not hard to compute and it is expressed through the Gauss hypergeometric 
function 2 -Pi- 

The problem becomes much more complicated when we look at x[^ . The basic 
reason is that this is the oscillatory zone for our orthogonal polynomials, and in 
the scaling limit that we need the period of oscillations tends to zero. Of course, 
one cannot expect to see any uniform convergence in this situation. 

Let us recall, however, that all we need is the asymptotics on the lattice. This 
remark is crucial. The way we compute the asymptotics on the lattice is, roughly 
speaking, as follows. We find meromorphic functions with poles in x\^} which 
coincide, up to a sign, with our orthogonal polynomials on X^ • These functions 
are also expressed through the 3-F2 series and look more complicated than the 
polynomials themselves. However, they possess a well-defined limit (convergence 
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with all the derivatives) which is again expressed through the Gauss hypergeometric 
function. This completes the computation of the asymptotics. 

The question is: how did we find these convenient meromorphic functions? The 
answer lies in the definition of the kernel K( N ' as L^ N \l + L^) -1 . It is not hard 
to see that the kernel L^ N > belongs to the class of (discrete) integrable operators, 
see [B3]. This implies that the kernel K^ N > can be expressed through a solution 
of a (discrete) Riemann-Hilbert problem (RHP, for short), see [B3, Proposition 
4.3]. It is the solution of this Riemann-Hilbert problem that yields the needed 
meromorphic functions. 

The problem of finding this solution explicitly requires additional efforts. The 
key fact here is that the jump matrix of this RHP can be reduced to a constant 
jump matrix by conjugation. It is a very general idea of the inverse scattering 
method that in such a situation the solution of the RHP must satisfy a difference 
(differential, in the case of continuous RHP) equation. Finding this equation and 
solving it in meromorphic functions yields the desired solution. 

It is worth noting that even though the correct formula for the limit correlation 
kernel K can be guessed from just knowing the Askey-Lesky orthogonal polyno- 
mials, the needed convergence of the kernels was only possible to achieve 
through solving the RHP mentioned above. 

Let us also note that computing the limit of the solution of our RHP is not 
completely trivial as well. The difficulty here lies in finding, by making use of 
numerous known transformation formulas for the 3F2 series, a presentation of the 
solution that would be convenient for the limit transition . 

(j) The main result. In (f) above we explained how to reduce our problem of 
harmonic analysis on U(oo) to the problem of computing the correlation func- 
tions of the process V. In this paper we prove that the nth correlation function 
p n (xi, . . . , x n ) of V has the determinantal form 



det[K(x i ,x j )] r i 



n 



1,2, 



Here K(x, y) is a kernel on M \ {±^} which can be written in the form 

F 1 (x)G 1 (y) + F 2 (x)G 2 (y) 



K(x,y) = 



x-y 



x,yGM\{±i}, 



where the functions Fi, Gi, F 2 , G 2 can be expressed through the Gauss hypergeo- 
metric function 2 F\. In particular, if x > \ and y > \ we have 



F 1 (x) = -G 2 (x) 



X \ x — 



sin(7rz) sin(7T2 / ) 

X + 



z + w', z' + w' 
z + z' + w + w' 



X 



G 1 (x) = F 2 (x) 



T(z + w + l)T(z + w' + l)T(z' + w + l)T(z' + w' + 1) 



T(z + z' + w + w' + l)T(z + z' + w + w' + 2) 

(££/+«,' + !) , lX H^i£ [ z + w , + 1 ^ z , + w , + 1 

z + z' + w + w' + 2 



x I .c — - ] I x + - ] 2 / ': 



— X 



A complete statement of the result can be found in Theorem 10.1 below. 
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(k) Symmetry of the kernel. The correlation kernel K(x,y) introduced above 
satisfies the following symmetry relations 



K(x,y) = 



K(y,x) if 

-K(y,x) if (|x| > |, \y\ < \) or < §, \y\ > \) . 



1 > 


1 

2 ' 


\y\ 


>\) 


or ( \x\ 


< 




<i 


1 > 


1 

2 ' 


\y\ 




or ( \x\ 


< 


i M 


>i 



Moreover, the kernel is real-valued. This implies that the restrictions of K to 
(— |, |) x (— i, |) and (R \ [— |, |]) x [— |, |]) are Hermitian kernels, while 
the kernel K on the whole line is a J-Hermitian 7 kernel. 

We have encountered certain J-Hermitian kernels in our work on harmonic anal- 
ysis on the infinite symmetric group, see [BOl], [P.I-V]. At that time we were not 
aware of the fact that examples of J-Hermitian correlation kernels had appeared 
before in works of mathematical physicists on solvable models of systems with posi- 
tive and negative charged particles, see [AF], [CJ1], [CJ2], [G], [Fl-3] and references 
therein. 

As was explained in (f), our system also contains 'particles of opposite charges'. 
The property of J-symmetry is closely related to this fact, see §5(f),(g) for more 
details. 

(1) Further development: Painleve VI. It is well known that for a deter- 
minantal point process with a correlation kernel K, the probability of having no 
particles in a region / is equal to the Fredholm determinant det(l — Kj), where 
Kj is the restriction of K to / x /. It often happens that such gap probability can 
be expressed through a solution of a (second order nonlinear ordinary differential) 
Painleve equation. One of the main results of [BD] is the following statement. 

Let K s be the restriction of the kernel K{x, y) of (j) above to (s, +oo) x (s, +oo). 
Set 

z + z' + w + w' z — z' + w — w' z — z' — w + w' 

vi = 2 ' U3 = 2 ' U4 = 2 ' 

, 2 n dlndet(l - K s ) 2 z/ 3 z/ 4 

*{*) = (* - 3 ) Js "is+sr- 

Then a(s) satisfies the differential equation 
-a' {{s 2 - I) a") 2 = (2 (so- 1 - a) a' - v\v^tf - (a' + v\)\o' + 4){a' + v\). 

This differential equation is the so-called a-form of the Painleve VI equation. We 
refer to [BD, Introduction] for a brief historical introduction and references on this 
subject. [BD] also contains proofs of several important properties of the kernel 
K{x,y) which we list at the end of §10 below. 

(m) Connection with previous work. In [BOl], [B02], [Bl], [B2], [B04] we 
worked out two other problems of harmonic analysis in the situations when spectral 
measures live on infinite-dimensional spaces. We will describe them in more detail 
and compare them to the problem of the present paper. 

The problem of harmonic analysis on the group S(oo) was initially formulated 
in [KOV]. It consists in decomposing certain 'natural' (generalized regular) unitary 



7 I.e., Hermitian with respect to the indefinite inner product defined by the matrix J = 
1 
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representations T z of the group S(oo) x S(oo), depending on a complex parameter 
z. In [KOV], the problem was solved in the case when the parameter z takes 
integral values (then the spectral measure has a finite-dimensional support). The 
general case presents more difficulties and we studied it in a cycle of papers [P.I-V] , 
[BOl-3], [Bl], [B2]. Our main result is that the spectral measure governing the 
decomposition of T z can be described in terms of a determinantal point process 
on the real line with one punctured point. The correlation kernel was explicitly 
computed, it is expressed through a confluent hypergeometric function (specifically, 
through the W-Whittaker function). 

The second problem deals with decomposition of a family of unitarily invariant 
probability measures on the space of all infinite Hermitian matrices on ergodic com- 
ponents. The measures depend on one complex parameter and essentially coincide 
with the measures {fi^ s '} mentioned in the beginning of (b) above. The problem 
of decomposition on ergodic components can be also viewed as a problem of har- 
monic analysis on an infinite-dimensional Cartan motion group. The main result of 
[B04] states that the spectral measures in this case can be interpreted as determi- 
nantal point processes on the real line with a correlation kernel expressed through 
a confluent hypergeometric function (this time this is the M-Whittaker function). 

These two problems and the problem that we deal with in this paper have a 
number of similarities. Already the descriptions of the spaces of irreducible objects 
(see Thoma [Th] for S(oo), Pickrell [Pi] and Olshanski-Vershik [OV] for measures 
on Hermitian matrices, and Voiculescu [Vo] for U(oo)) are quite similar. Fur- 
thermore, all three models have some sort of an approximation procedure using 
finite-dimensional objects, see [VK1], [OV], [VK2], [OkOl]. The form of the corre- 
lation kernels is also essentially the same, with different special functions involved 
in different problems. 

It is worth noting that the similarity of theories for the two groups S(oo) and 
U(oo) seems to be a striking phenomenon. In addition to mentioned above, it 
can be traced in the geometric construction of the 'natural' representations and in 
probabilistic properties of the corresponding point processes. At present we cannot 
completely explain the nature of this parallelism (it looks quite different from the 
well-known classical connection between the representations of the groups S(n) and 
U(N)). 

However, the differences between all these problems should not be underesti- 
mated. Indeed, the problem of harmonic analysis on £(00) is a problem of as- 
ymptotic combinatorics consisting in controlling the asymptotics of certain explicit 
probability distributions on partitions of n as n — > 00. One consequence of such as- 
ymptotic analysis is a simple proof and generalization of the Baik-Deift- Johansson 
theorem [BDJ] on longest increasing subsequences of large random permutations, 
see [BOO] and [B03]. The problem of decomposing measures on Hermitian matri- 
ces on ergodic components is of a different nature. It belongs to Random Matrix 
Theory which deals with asymptotics of probability distributions on large matrices. 
In fact, for a specific value of the parameter, the result of [B04] reproduces one of 
the basic computations of Random Matrix Theory — that of the scaling limit of 
the Dyson's circular ensemble. 

The problem solved in the present paper is more general comparing to both 
problems described above. Our model here depends on a larger number of param- 
eters, it deals with a more complicated group and representation structure, and 
the analysis requires substantial amount of new ideas. Moreover, in appropriate 
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limits this model degenerates to both models studied earlier. The limits, of course, 
are very different. On the level of correlation kernels this leads to two different 
degenerations of the Gauss hypergeometric function to confluent hypergeometric 
functions. We view the U (oo)-model as a unifying object for the combinatorial 
and random matrix models, and we think that it sheds some light on the nature of 
the recently discovered remarkable connections between different models of these 
two kinds. 

The model of the present paper can be also viewed as the top of a hierarchy 
of (discrete and continuous) probabilistic models leading to determinantal point 
processes with 'integrable' correlation kernels. In the language of kernels this looks 
very much like the hierarchy of the classical special functions. A description of the 
£ 5'(oo)-part' of the hierarchy can be found in [B03]. The subject of degenerating 
the U (oo)-model to simpler models (in particular, to the two models discussed 
above) will be addressed in a later publication. 

(n) Organization of the paper. In Section 1 we give a brief introduction to 
representation theory and harmonic analysis of the infinite-dimensional unitary 
group U(oo). Section 2 explains how spectral decompositions of representations 
of U(oo) can be approximated by those for finite-dimensional groups U(N). In 
Section 3 we introduce a remarkable family of characters of U(oo) which we study 
in this paper. In Section 4 we reformulate the problem of harmonic analysis of these 
characters in the language of random point processes. Section 5 is the heart of the 
paper: there we develop general theory of discrete determinantal point processes 
which will later enable us to compute the correlation functions of our concrete 
processes. In Section 6 we show that the point processes introduced in Section 4 
are determinantal. In Section 7 we derive discrete orthogonal polynomials on Z 
with the weight function (0.7). This allows us to write out a correlation kernel 
for approximating point processes associated with £/(iV)'s. Section 8 is essentially 
dedicated to representing this correlation kernel in a form suitable for the limit 
transition N — > oo. The main tool here is discrete Riemann-Hilbert problem. 
Section 9 establishes certain general facts about scaling limits of point processes 
associated with restrictions of characters of U(oo) to U(N). The main result here is 
that an appropriate scaling limit yields the spectral measure for the initial character 
of U(oo). In Section 10 we perform such a scaling limit for our concrete family of 
characters. Section 11 describes a nice combinatorial degeneration of our characters. 
In this degeneration the spectral measure loses its infinite-dimensional support and 
turns into a Jacobi polynomial ensemble. Finally, Appendix contains proofs of 
transformation formulas for the hypergeometric series 3F2 which were used in the 
computations. 
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served as a Clay Mathematics Institute Long-Term Prize Fellow. 

14 



1. Characters of the group U(oo) 

(a) Extreme characters. Let U(N) be the group of unitary matrices of order 
N. For any N > 2 we identify U(N — 1) with the subgroup in U(N) fixing the iVth 
basis vector, and we set 

U(oo) = h^U(N). 

One can view U(oo) as a group of matrices U = [Uij]fj =1 such that there are 
finitely matrix elements Uij not equal to Sij, and U* = U~ x . 

A character of U(oo) is a function x : U(oo) — > C which is constant on conjugacy 
classes, positive definite, and normalized at the unity (%(e) = 1). We also assume 
that x is continuous on each subgroup U(N) C U(oo). The characters form a convex 
set. The extreme points of this convex set are called the extreme characters. 

A fundamental result of the representation theory of the group U(oo) is a com- 
plete description of extreme characters. To state it we need some notation. 

Let R°° denote the product of countably many copies of R, and set 

R 4oo+2 = R oo x R oo x R oo x R oo x R x R _ 

Let O C R 4oo+2 be the subset of sextuples 
such that 

a ± = (af > a± > ■ ■ ■ > 0) G [3 ± = (pf > 0} > ■ ■ ■ > 0) e R°°, 

oo 

Set 

oo 

i=i 

and note that 7 + ,7~ are nonnegative. 

To any w£Q we assign a function on U(oo): 

s n ( u<~w ■-'n i:gi::ij::g::::ij }. 

ueSpectrum(C7) L i=l * v y 1 v ; J 

Here £7 is a matrix from U(oo) and w ranges over the set of its eigenvalues. All 
but finitely many u's equal 1, so that the product over u is actually finite. The 
product over i is convergent, because the sum of the parameters is finite. Note also 
that different u/s correspond to different functions: here the condition f3^ < 1 

plays the decisive role, see [013, Remark 1.6]. 

Theorem 1.1. The functions x > where uj ranges overfl, are exactly the extreme 
characters of the group U(oo). 

Proof. The fact that any is an extreme character is due to Voiculescu [Vo]. 
The fact that the extreme characters are exhausted by the X s can be proved in 
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two ways: by reduction to an old theorem due to Edrei [Ed] (see [Boy] and [VK2]) 
and by Vershik-Kerov's asymptotic method (see [VK2] and [OkOl]). □ 

The coordinates af , ftf, and ^ (or 5^) are called the Voiculescu parameters 
of the extreme character x^ • Theorem 1.1 is similar to Thoma's theorem which 
describes the extreme characters of the infinite symmetric group, see [Th, VK1, Wa, 
KOO]. Another analogous result is the classification of invariant ergodic measures 
on the space of infinite Hermitian matrices (see [OV] and [Pi2]). 

(b) Spectral measures. Equip IR 4oo+2 with the product topology. It induces a 
topology on O. In this topology, O is a locally compact separable space. On the 
other hand, we equip the set of characters with the topology of uniform convergence 
on the subgroups U(N) C U(oo), N = 1, 2, . . . . One can prove that the bijection 
uj < — ► x is a homeomorphism with respect to these two topologies, see [013, §8]. 
In particular, x (U) is a continuous function of u for any fixed U G U(oo). 

Theorem 1.2. For any character x of the group U(oo) there exists a probability 
measure P on the space O such that 

X(U)= [ x (aj) (U)P(dw), UEU(oo). 
Jo. 

Such a measure P is unique. The correspondence x ^ -P m fl bijection between the 
set of all characters and the set of all probability measures on O. 

Here and in what follows, by a measure on Vt we mean a Borel measure. We call 
P the spectral measure of x- 

Proof. See [013, Theorem 9.1]. □ 

Similar results hold for the infinite symmetric group (see [KOO]) and for invariant 
measures on infinite Hermitian matrices (see [B04]). 

(c) Signatures. Define a signature A of length N as an ordered sequence of inte- 
gers with N members: 

A = (Ai > A 2 > ••• > X N I A, G Z). 

Signatures of length N are naturally identified with highest weights of irreducible 
representations of the group U(N), see, e.g., [Zh]. Thus, there is a natural bijection 
A < — >■ x X between signatures of length iV and irreducible characters of U(N) (here 
we use the term "character" in its conventional sense). The character x X can De 
viewed as a rational Schur function (Weyl's character formula) 

A/ \ detV U,j=i,...,N 

X ( Ul ,...,u N )= . 

det [^ J«,i=i JV 

Here the collection (m, . . . , u n ) stands for the spectrum of a matrix in U(N). 

We will represent a signature A as a pair of Young diagrams (A + ,A~): one 
consists of positive Aj's, the other consists of minus negative A^'s, zeros can go in 
either of the two: 

A = (X 1 , A 2 , . . . , — A 2 , X± ). 
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Let d + = d(X) and d~ = d(X~), where the symbol d{ ■ ) denotes the number of 
diagonal boxes of a Young diagram. Write the diagrams A + and A~ in Frobenius 
notation: 

We recall that the Frobenius coordinates pi, qi of a Young diagram v are defined 
by 

Pi = i/i-i, q i = (v') i -i, i = l,...,d(v), 

where v' stands for the transposed diagram. Following Vershik-Kerov, we introduce 

the modified Frobenius coordinates of v by 

pi = Pi + 5, qi = qi + \- 

Note that YliPi + Qi) = IH> where \v\ denotes the number of boxes in v. 
We agree that 

Pi = Qi = 0, i>d{v). 

(d) Approximation of extreme characters. Recall that the dimension of the 
irreducible representation of U(N) indexed by A is given by Weyl's dimension for- 
mula 

Dim JV A = X A (l,...,l)= II At ~ Z -I- J+J - 

" ^ ' i<i<j<N 3 1 

Define the normalized irreducible character indexed by A as follows 

DimAr A 

Clearly, x A (e) = 1. 

Given a sequence {/jv}jv=i,2,... of functions on the groups U(N), we say that 
/jv's approximate a function / defined on the group U(oo) if, for any fixed N = 
1,2,..., the restrictions of the functions /jv (where A" > N ) to the subgroup U(N ) 
uniformly tend, as N — > 00, to the restriction of / to U(N ). 

Theorem 1.3. Any extreme character x of U (00) can be approximated by a se- 
quence °f normalized irreducible characters of the groups U(N). 

In more detail, write x = X^ N K where {A(A/")}jv=i,2,... is a sequence of 
signatures, and letp^(N) and q^(N) stand for the modified Frobenius coordinates 
of (X(N)) ± . Then the functions x^ approximate x if an d only if the following 
conditions hold 

PtW_+ fW_ fl± l( A ( iV )) ± l _ c± 



lim — = a , lim — = 3 , lim 

»r .„ AT I ' »r _„ AT f I ' »7 .. 



where i = 1, 2, ... , and af, df, S ± are the Voiculescu parameters of the character 
X- 

This claim reveals the asymptotic meaning of the Voiculescu parameters. Note 
that for any u> = (a + , (3 + ; a~ , (3~; 5 + , S~) G O, there exists a sequence of signatures 
satisfying the above conditions, hence any extreme character indeed admits an 
approximation. 

Proof. This result is due to Vershik and Kerov, see their announcement [VK2]. A 
detailed proof is contained in [OkOl] . □ 

For analogous results, see [VK1, OV]. 
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2. Approximation of spectral measures 

(a) The graph GT. For two signatures v and A, of length N — 1 and N, respec- 
tively, write v ■< A if 

Ai > v\ > A2 > ^2 > • • • > ^jv-i > Atv • 

The relation z/ -< A appears in the Gelfand-Tsetlin branching rule for the irreducible 
characters of the unitary groups, see, e.g., [Zh]: 

X X (u 1 ,...,u N - 1 ,l) = ^ x v ■ 

The Gelfand-Tsetlin graph GT is a Z_|_ -graded graph whose iVth level GT/v 
consists of signatures of length N. Two vertices v G GT/v_i and A G G1V are 
joined by an edge if v -< A. This graph is a counterpart of the Young graph 
associated with the symmetric group characters [VK1], [KOO]. 

(b) Coherent systems of distributions. For v G GTjv_i and A G GT/v, set 

{ DiniAr-i v ^ 
DimAr A 
0, v -/< A. 

This is the cotransition probability function of the Gelfand-Tsetlin graph. It satis- 
fies the relation 

q(v,X) = l, VAgGT^v. 

^6GTiv_i 

Assume that for each iV = 1,2,... we are given a probability measure Pjv on 
the discrete set GTat. Then the family {Pn}n=i,2,... is called a coherent system if 

Pjv-iH= ?(^A)Pjv(A), VAT = 2,3,..., Vi/gGTjv-l 

AeGTjv 

Note that if Pjv is an arbitrary probability measure on GTjv then this formula 
defines a probability measure on GHV_i (indeed, this follows at once from the 
above relation for q(v, A)). Thus, in a coherent system {Pjv}jv=i,2,..., the ATth term 
is a refinement of the (N — l)st one. 

Proposition 2.1. There is a natural bijective correspondence x i — ^ {Pv} between 
characters of the group U (00) and coherent systems, defined by the relations 

x\u(N)= J2 P ^Wx\ iV = l,2,.... 

AeGTjv 

Proof See [013, Proposition 7.4]. □ 

A similar claim holds for the infinite symmetric group S(oo), see [VK1], [KOO], 
and for the infinite-dimensional Cartan motion group, see [OV]. Note that {Pat} 
can be viewed as a kind of Fourier transform of the corresponding character. 

The concept of a coherent system {P/v} is important for two reasons. First, we 
are unable to calculate directly the "natural" nonextreme characters but we dispose 
of nice closed expressions for their "Fourier coefficients" Pjv(A), see the next section. 
Note that in the symmetric group case the situation is just the same, see [KOV], 
[BOl-3]. Second, the measures P/v approximate the spectral measure P, see below. 
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(c) Approximation P N — > P. Let x be a character of U(oo) and let P and {Pn} 
be the corresponding spectral measure and coherent system. 

For any N = 1, 2, . . . , we embed the set GTjv into O C M 4oo+2 as follows 



GT N 3 A i — ► (a + ,6 + ;a",6" 



;c + ,c~ 



~± 
Pi_ 

N ' 



6? 



AT 



4oo+2 



where i = 1,2,..., and p^ 1 , qf 1 are the modified Frobenius coordinates of A ± . 

Let P_ N be the pushforward of Pn under this embedding. Then P_ N is a proba- 
bility measure on O. 

Theorem 2.2. As N — > oo, t/ie measures P N weakly tend to the measure P. I.e., 
for any bounded continuous function F on Vt, 

lim (F,P N ) = (F,P). 

Proof. See [013, Theorem 10.2]. □ 

This result should be compared with [KOO, Proof of Theorem B in §8] and [B04, 
Theorem 5.3]. Its proof is quite similar to that of [B04, Theorem 5.3]. 

Theorem 2.2 shows that the spectral measure can be, in principle, computed if 
one knows the coherent system {Pn}- 

3. ZW-MEASURES 

The goal of this section is to introduce a family of characters x °f the group 
U(oo), for which we solve the problem of harmonic analysis. We describe these 
characters in terms of the corresponding coherent systems {Pn}- For detailed 
proofs we refer to [013]. 

Let z, z', w, w' be complex parameters. For any N = 1, 2, . . . and any A G GTn 

set 

P' N {\ | z, z'w, w') = Dim^(A) 



y F(z - A, + i)T(z' - A, + i)T(w + N + 1 + X t - i)T(w' + N + 1 + A, - i) ' 

where Dini/v A was defined in §1. Clearly, for any fixed iV and A, P N (^ I z > z> i w i w> ) 
is an entire function on C 4 . Set 

V = {(z, z', w, w') e C 4 I U{z + z' + w + w') > -1}. 

This is a domain in C 4 . 

Proposition 3.1. Fix an arbitrary N = 1,2, . . . . The series of entire functions 

J2 P' N {\\z,z',w,w') 

AeGTjv 
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converges in the domain V, uniformly on compact sets. Its sum is equal to 

c / / a TT r(z + z'+w + w' + z) 

^ ™' W ' Al F ( z + w + i) T (z + w' + i)T(z> + w + i)T{z> +w' + i)r(i) 

Proo/. See [013, Proposition 7.5]. □ 

Note that in the special case N = 1, the set GTi is simply Z and the identity 

P((A | 2;', tu, «/) = Si(z, z' , w, w') 

AeGTi 

is equivalent to the well-known DougalPs formula (see [Er, vol 1, §1-4]). 
Consider the subdomain 

V = {(z, z', w,w') eV | z + w, z + w', z' + w,z' + w'^ -1, -2, . . . } 
= {(z, z' , w, w') G V | Sn(z, z' , w, w') ^ 0}. 
For any (z, z', w, w') G V we set 

P N (X\z,z',w,w') = P ^ (A ' V'^a . iV = l,2,..., AGCT W . 
Then, by Proposition 3.1, 

Pjv(A | 2;, z',w,w') = 1, (2;, 2/, ty, u/) G "Do 7 

AeGTjv 

uniformly on compact sets in T>q. 

Proposition 3.2. Let (z, z' ,w,w') G Vq. For any N = 2,3,..., the coherency 
relation of§2(b) is satisfied, 

Pn-i(v \ z, z' ,w,w') = q{y, A)P(A | z, z' , w, w'). 

Proo/. See [013, Proposition 7.7]. □ 

Combining this with Proposition 2.1 we conclude that {Pat(- | z, z' ,w,w')}, 
where N = 1, 2, . . . , is a coherent system provided that (2, 2', to, «/) G Do satisfies 
the positivity condition: for any A/" = 1, 2, ... , the expression P^-(A | 2, 2', w, w') is 
nonnegative for all A G GTat. (Note that there always exists A for which P' N (X \ 
z, z', w, w') 7^ 0, because the sum over A's is not 0.) We proceed to describing a set 
of quadruples (z, z', w, w') G V Q satisfying the positivity condition. 

Define the subset Z C C 2 as follows: 

2> = -Zprinc U -2^compl U ^degen; 

2 P rinc = {(^^)eC 2 \M 2 |z / = ^}, 

^ C ompi = {(-2, z') G 1R 2 I 3m G Z, m < 2, z < m + 1}, 

^degen,m = {(-2, z') G R 2 \ Z = 771, Z > 771 — 1, Or Z = 771, Z > 771 — 1}, 

where "princ", "compl", and "degen" are abbreviations for "principal", "comple- 
mentary" , and "degenerate" , respectively. For an explanation of this terminology, 
see [013]. 
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Proposition 3.3. Let (z,z') G C 2 . 

(i) The expression (T(z — k)T(z' — k))~ x is nonnegative for all k G Z if and only 
if(z,z')eZ. 

(ii) If (z, z') G 2 P rinc U ^compi #&en expression is strictly positive for all 
keZ. 

(iii) 7/ (z, z') G ^degen,™, #&en t/izs expression vanishes for k = m, m + 1, . . . and 
is strictly positive for k = m — 1, m — 2,... . 

Proo/. See [013, Lemma 7.9]. □ 

Definition 3.4. The set o/ admissible values of the parameters z, z' ,w,w' is the 
subset P a dm C D of quadruples (z, z' ,w,w') such that both (z,z') and (w,w') 
belong to Z. When both (z, z') and (w, w') are in Z, an extra condition is added: 
let k, I be such that (z,z') G ^degen,fc and (w,w') G ^degen,z; then we require 
k + 1 > 0. A quadruple (2, 2', w;, u/) will be called admissible if it belongs to the set 

^adm- 

Note that in this definition we do not assume a priori that (z, z', w, w') belongs 
to the subdomain Vq C V. However the conditions imposed on (z, z' ,w,w') imply 
that £>adm C 1*0 ? see below. 

Proposition 3.5. Let (z, z' ,w,w') G X\dm and let N = 1,2,... . Xnen P^(A | 
z, z' ,w,w') > /or any A G GT/v, and there exists A G GTn for which the above 
inequality is strict. 

Proof. The first claim follows from Proposition 3.3 (i). Now we shall describe the 
set of those A G GT/v for which P' N (X | z, z', w, w') > 0. 

When both (z, z') and (w, w') are in 2 pr inc U Z com pi then, by Proposition 3.3 (ii), 
this is the whole GTjv- 

When (w, w') G Z princ U Z couip i and (z, z') G Z, say, (z, z') G 2de g en,m, then this 
set is formed by A's satisfying the condition Ai < m. Indeed, this readily follows 
from claims (ii) and (iii) of Proposition 3.3. 

Likewise, when (z, z') G Z princ U Z comp \ and (w, w') G ^degen,m, then the condi- 
tion takes the form A^v > — m. 

Finally, when both (z, z') and (w, w') are in Z, say (z, z') G Za e gen,k and (w, w') G 
^degen,;, then the set in question is described by the conditions Ai < k, Xn > — I- 
The set is nonempty provided that k > —I, which is exactly the extra condition 
from Definition 3.4. 

Note that if k = — I then this set consists of a single element A = (k, . . . , k). □ 

Proposition 3.5 implies that £> a dm C Vq. Of course, this can be checked directly, 
but the claim is not entirely obvious, for instance, when both (z, z') and (w, w') are 
in Z corap \. 

Now we can summarize the above definitions and results in the following theorem. 

Theorem 3.6. For any admissible quadruple (z, z' ,w,w'), the family {Pjv( - 
z, z' ,w,w')}, where N = 1,2,..., is a coherent system, so that it determines a 
character Xz,z', w,w' of the group U(oo). 

Proof. Indeed, let (z, z' , w, w') be admissible. Since (z, z', w, w') is in Vq, the 
definition of P/v(A | z, z' , w, w/)'s makes sense. By Proposition 3.5, for any N, 
Pjv(- I z, z' ,w,w') is a probability distribution on G1V. By Proposition 3.2, the 
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family {Pn(- I z, z' , w, w')}n=i,2,... is a coherent system. By Proposition 2.1, it 
defines a character of U(oo). □ 

Remark 3.7. The set of characters of the form Xz,z',w,w' is stable under tensoring 
with one-dimensional characters (det(-)) fc , where k E Z. Indeed, the sets V, V , 
and "Dadm are invariant under the shift 

(z, z' , w, w') 1— > (2; + fc, z' + k, w — k, w' — k), 

and we have 

-P/v(A + (fc, . . . , fc) I z, 2/, iu, «/) = Pat (A | -2 + fc, z 7 + fc, -u; — k, w' — k). 

On the other hand, in terms of coherent systems, tensoring with (det( • )) k is equiv- 
alent to shifting A by (fc, . . . , k). 

Remark 3.8. In the special case when both (z, z') and (w,w') are in Zdegen, a 
detailed study of the distributions Pjv(- | z, z' ,w,w') from a combinatorial point 
of view was given by Kerov [Ke]. 

Remark 3.9. As we see, the structure of the set of all admissible parameters is 
fairly complicated. However, all the major formulas that will be obtained below do 
not feel this structure; they hold for all admissible parameters. The explanation 
of this phenomenon is rather simple: the quantities in question (like correlation 
functions) can usually be defined for the parameters varying in domain which is 
much larger than "D a dm, see e.g. Propositions 3.1 and 3.2 above. Thus, the formulas 
for these quantities usually hold on an open subset of C 4 containing P a dm- It is only 
when we require certain quantities to be positive in order to fit our computations 
in the framework of probability theory, we need to restrict ourselves to the smaller 
set of admissible parameters. 

4. TWO DISCRETE POINT PROCESSES 

In this section we will explain two different ways to associate to the measure Pn 
introduced in the previous section a discrete point process. We also show how the 
two resulting processes can be obtained one from the other. 

First, we recall the general definition of a random point process. 

Let X be a locally compact separable topological space. A multiset X in X is 
a collection of points with possible multiplicities and with no ordering imposed. 
A locally finite point configuration {configuration, for short) is a multiset X such 
that for any compact set A C X the intersection X n A is finite (with multiplicities 
counted) . This implies that X itself is either finite or countably infinite. 

The set of all configurations in X is denoted by Conf(X). Given a relatively 
compact Borel set A C X, we introduce a function Ma on Conf(X) by setting 
Ma(X) = \X n A\. We equip Conf(X) with the Borel structure generated by all 
functions of this form. 

A random point process on X (point process, for short; another term is random 
point field) is a probability Borel measure V on the space Conf(X). 

We do not need the full generality of these definitions in this section. Here the 
situation is rather simple: all our processes are discrete (that is, the space X is 
discrete), and the point configurations are finite. However, in §9 we will consider a 
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continuous point process with infinitely many particles, and then we will need the 
above definitions. 
Consider the lattice 



and divide it into two parts 



Z, N is odd, 

Z + i, N is even, 



X Xj n L-l Xout 3 
*».-< 2 , 2 2 , 2 j>, |X in |-JV, 

iV + 3 iV + ll fN + 1 N + 3 \ 
Mini — <i • ■ • • ^ ' 2 I ^ 1 2 ' 2 ' I ' out l ~~ °°' 

Let pi = —i, i = 1, . . . , N. For any A G GTn we set 

£(A) = {Ai + pi, . . . , Ajv + p N }- 

Clearly, A i— > £(A) defines a bijection between GUV and the set of iV-point multi- 
plicity free configurations on X. 

Now we define another correspondence between signatures and point configura- 
tions. Let us represent a signature A as a pair of Young diagrams (A + , A~), see 
§l(c). 

Finally, we define a point configuration as 

wu r , iv + n f n — l ,} r _ n + i\ r N-i i 

(4.1) 

where i = and j = 1 , . . . , , see §l(c) for the notation. Note that if 

A = then the configuration is empty. 
From the inequalities 

p+ > ■ ■ ■ > p+ + > 0, (?+>•••> > 0, 
> ■ ■ ■ > p~_ > 0, ? r > • • • > q~_ > 0, 
rf+ + cT < iV 

it follows that X(A) consists of an even number of distinct points (equal to 2(d + + 
<i~)), of which a half lies in X ou t while another half lies in X; n . Finite point config- 
urations with this property will be called balanced. 

Conversely, each balanced, multiplicity free configuration on X is of the form 
X(X) for one and only one signature A G GIV- Thus, the map A i— > £(A) defines a 
bijection between GTjy and the set of finite balanced configurations on X with no 
multiplicities. 

Define an involution on the set Conf(X) of multiplicity free point configurations 
on X by 

x i-> x A = x a x in = (x n x out ) u (X in \ x). 

Since |Xi n | = N, this involution defines a bijection between iV-point configurations 
and finite balanced configurations. 
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Proposition 4.1. In the above notation, X(X) = £(A) A for any signature A G 
GT N . 



For instance, let iV = 7 and A = (4, 2, 2, 0, -1, -2, -2). Then 



£(A) = {7,4,3,0,-2,-4,-5} 



and, since Xi n = 



{3,2,1,0,-1,-2, 



3}, we have 



X(X) = C(X) 



A 



{7,4,2,1,-1,-3,-4,-5}. 



On the other hand, A+ = (4, 2, 2) = (3, | 2, 1), A" = (2, 2, 1) = (1, | 2, 0), and 
(4.1) gives the same X(X). 

Another example: for the zero signature we have £(0) = Xi n and X(Q) = 0. 

Proof of Proposition 4-1- Geometric constructions described below are illustrated 
by a picture at the end of the text. Consider a plane with Cartesian coordinates 
(x, y) and put the lattice X = %( N ^ on the vertical = 0, so that each a G X 

is identified with the point (0, a) of the plane. Draw a square grid in the plane, 
formed by the horizontal lines y = a+ ^, where a ranges over X, and by the vertical 
lines x = b, where b ranges over Z. We represent A by an infinite polygonal line L 
on the grid, as follows. 

Denote by Aq, . . . , An the horizontal lines defined by y = ^ , y = ^ — 1, 
y = — ^r, respectively. We remark that these lines belong to the grid: indeed, X 
coincides with Z shifted by ^f^-, so that the points -1,...,— | belong to 
the shift of X by \. 

The polygonal line L first goes along Aq, from right to left, starting at x = +oo, 
up to the point with the coordinate x = X±. Then it changes the direction and goes 
downwards until it meets the next horizontal line A\. Then it goes along Ai, again 
from right to left, up to the point with the coordinate x = A2, etc. Finally, after 
reaching the lowest line An at the point with the coordinate x = Xn, it goes only 
to the left, along this line. 

Further, we define a bijective correspondence a <-> s between the points a G X 
and the sides s of L, as follows. Given a, we draw the line x + y = a, it intersects 
L at the midpoint of a side, which is, by definition, s. Let us call a a "w-point" or 
a "/i-point" according to whether the corresponding side s is vertical or horizontal. 
Thus, the whole set X is partitioned into "^-points" and "/i-points" . 

The "f -points" of X are exactly those of the configuration C(X). Consequently, 
the collection (C(X) H X out ) U (Xi n \ £(A)) is formed by the 'Vpoints" from X ou t 
and the "/i-points" from Xi n - 

On the other hand, the correspondence a <-> s makes it possible to interpret the 
same collection of points in terms of the Frobenius coordinates of the diagrams A + 
and A - . Indeed, the diagram A+ can be identified with the figure bounded by the 
horizontal line Aq, the vertical line x = 0, and by L. Then the line x + y = ^ 
coincides with the diagonal of A + . Above this line, there are d + vertical sides 
of L, say, s±, . . . , which lie in the rows of A + with numbers 1, . . . , d + . The 
corresponding Frobenius coordinates are = Xi — i, where % = 1, . . . , d + . It easily 
follows that the midpoint of the side Sj lies on the line x + y = +p^, i.e., Si 
corresponds to +p^~- In this way we get the first component [p'l + ^p-} C 
X(X), see (4.1). The remaining three components are interpreted similarly. □ 
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Fix any admissible quadruple (z, z' , w, w') of parameters and consider the cor- 
responding probability measure Pn on GTjv, see §3. Taking the pushforwards of 
the measure Pn under the maps A i— > £(A) and A h- > X(X) we get two point pro- 
cesses on the lattice X, which we denote by J>( N > and V^ N \ respectively. We are 
mainly interested in the process V^ N \ which is defined by A i— > X(X); the process 
T>( N > defined by A i— > £(A) will play an auxiliary role. Proposition 4.1 implies that 
■p( N )(X) = V {N) (X A ) for any finite configuration X. 

5. Determinantal point processes. General theory 

(a) Correlation measures. Let V be a point process on X (see the definition in 
the beginning of §4) , and let A denote an arbitrary relatively compact Borel subset 
of X. Then Ma is a random variable with values in {0, 1,2,...}. We assume that 
for any A as above, Ma has finite moments of all orders. 

Let n ranges over {1,2,...}. The nth correlation measure of V, denoted as p n , 
is a Borel measure on X n , uniquely defined by 

p n (A n ) = E[Ma(M a -l)...(M A -n+l)}, 

where the symbol E means expectation with respect to the probability space (Conf (X), 
Equivalently, for any bounded compactly supported Borel function F on I", 



(F, Pn) = [ 



XeConf(X) 



/ \ 

F(xi,...,x„) 

pairwise distinct / 



V(dX), 



where the summation is taken over all ordered n-tuples of pairwise distinct points 
taken from the (random) configuration X (here a multiple point is viewed as a 
collection of different elements). 

The measure p n takes finite values on the compact subset of X n . The measure 
p n is symmetric with respect to the permutations of the arguments. 

Under mild assumptions about the growth of p n (A n ) as n — > oo (here A is an 
arbitrary compact set), the collection of the correlation measures pi, p2, ■ ■ ■ defines 
the initial process V uniquely. See [Len] and [So, (1.6)]. 

When there is a "natural" reference measure u on X such that, for any n, p n is 
absolutely continuous with respect to the product measure u® n , the density of p n is 
called the nth correlation function. For instance, this always holds if the space X is 
discrete: then as u one takes the counting measure on X. The correlation functions 
are denoted as p n (x\, . . . , x n ). 

If the space X is discrete and the process is multiplicity free then p n (xi, . . . , x n ) is 
the probability that the random point configuration contains the points x\, . . . ,x n 
(here a^'s are pairwise distinct, otherwise p n (xi, • • • , x n ) = 0). 

For a general discrete process, p n (xi, • • • , x n ) is equal to the sum of weights of the 
point configurations with certain combinatorial prefactors computed as follows: if 
x has multiplicity k in the multiset (xi, . . . , x n ) and has multiplicity m in the point 
configuration in question, then this produces the prefactor m(m — 1) • • • (m — k + 1) 
(such prefactor is computed for every element of the set {x±, . . . , x n }). Note that 
this prefactor vanishes unless m > k for every x G {xi, . . . , x n }. 
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(b) Determinantal processes. A point process is called determinantal if there 
exists a function K(x, y) on X x X such that, for an appropriate reference measure 
/i, the correlation functions are given by the determinantal formula 



p n {x u . . . , x n ) = det[K(xi, Xj)]? j=1 , n = 1, 2, . . . . 

The function K is called the correlation kernel of the process. It is not unique: 
replacing K(x,y) by f(x)K(x,y)f(y)~ 1 , where / is an arbitrary nonzero function 
on X, leaves the above expression for the correlation functions intact. 

If the reference measure is multiplied by a positive function / then the correlation 
kernel should be appropriately transformed. For instance, one can multiply it by 
(./VIA//)) ; 

It is often useful to view K(x,y) as the kernel of an integral operator acting in 
the Hilbert space L 2 (X, fi). We will denote this operator by the same symbol K. 

Assume that a function K{x, y) is Hermitian symmetric (i.e., K{x, y) = K(y, x)) 
and locally of trace class (i.e., its restriction to any compact set A C X defines a 
trace class operator in L 2 (A, //), where // is a fixed reference measure). Then K(x, y) 
is the correlation kernel of a determinantal point process if and only if the operator 
K in L 2 (X, /u) satisfies the condition < K < 1, see [So]. However, there are 
important examples of correlation kernels which are not Hermitian symmetric, see 
below. 

If X is a discrete countably infinite space then any multiset with finite multi- 
plicities is a configuration. As \x we will always take the counting measure. A 
correlation kernel is simply an infinite matrix with the rows and columns labeled 
by the points of X. For any determinantal process on X the random configuration 
is multiplicity free with probability 1. Indeed, if any two arguments of the nth 
correlation function coincide then the defining determinant above vanishes. 

(c) The complementation principle. Assume that X is discrete and fix a subset 
Z C X. For a subset X in X let XAZ denote its symmetric difference with Z, i.e., 
XAZ = (Xr\Z)U(Z\X), where Z = X \ Z. The map X ^ XAZ, which we will 
denote by the symbol A, is an involution on multiplicity free configurations. If the 
process V lives on the multiplicity free configurations, we can define its image V A 
under A. 

Assume further that V is determinantal and let K be its correlation kernel. Then 
the process V A is also determinantal. Its correlation kernel K A can be obtained 
from K as follows: 

a N f K(x, -y), x E Z, 

K A (x,y) = { V ' \ (5.1) 
[ 5 xy - K(x,y), x E Z, 

where 5 xy is the Kronecker symbol. See [BOO, §A.3]. 
Note that one could equally well use the formula 



K A (x,y) = 



K(x,y), yeZ, 
5 xy -K(x,y), y E Z, 



obtained from (5.1) by multiplying the kernel by the function e(x)e(y), where s{ • ) 
is equal to 1 on Z and to —1 on Z. This operation does not affect the correlation 
functions, see §5(b). 
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We call the passage from the process V to the process V , together with formula 
(5.1) the complementation principle. The idea was borrowed from unpublished work 
notes by Sergei Kerov connected with an early version of [BOO]. 

Note that Proposition 4.1 can now be restated as follows: 

where the role of the set Z is played by Xi n . 

(d) Discrete polynomial ensembles. Here we assume that X is a finite or count- 
ably infinite subset of R without limit points. 

Assume that we are given a nonnegative function f(x) on X. Fix a natural 
number N. We consider / as a weight function: denoting by fi the counting measure 
on X we assign to / the measure f/i on X. 

We impose on / two basic assumptions: 

(*) / has finite moments at least up to order 2N — 2, i.e., 

x 2N ~ 2 f{x)< 00. 

xEX 

(**) / does not vanish at least at N distinct points. 

Under these assumptions the functions 1, x, . . . , x N ~ x on X are linearly indepen- 
dent and lie in the Hilbert space L 2 (X, ffi). Let po = 1, p±, . . . , Pn-i be the monic 
polynomials obtained by orthogonalizing the system (1, x, . . . , x N_1 ) in L 2 (X, ffi). 

We set 

K = (Pn,Pn)L2(X,/ M ) = ^Pn(x)f(x), 71 = 0, . . . , iV - 1, 

and consider the Christoffel-Darboux kernel 

^ p»WpM , x , y eX. 

n=0 hn 

This kernel defines an orthogonal projection operator in L 2 (X, ffi); its range is the 
iV-dimensional subspace spanned by 1, x, . . . , x N ~ l . 

Consider an isometric embedding L 2 (X, ffi) — > £ 2 (X) which is defined as multi- 
plication by a//( • )• Under this isomorphism the Christoffel-Darboux kernel turns 
into another kernel which we will call the normalized Christoffel-Darboux kernel 
and denote as K CD : 

K CD (x, y) = v / 7R7M ■ E PniX l Pniy) , ^yeX. (5.2) 

n=0 ^ 

This kernel defines a projection operator in £ 2 (X) of rank N. 

Let Conf;v(X) denote the set of iV-point multiplicity free configurations (subsets) 
in X. For X G Confer (X) we set 

v 2 (x)= n ( Xi -xj) 2 , 

l<i<j<N 
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where x±, . . . ,xn are the points of X written in any order. 
Under the assumptions (*) and (**) we have 



E (\{f{x)-V\X)\<^. 

JonfjvfX) \xex J 



< 

XeConfjv(X) 

Therefore, we can form a point process on X which lives on Confer (X) and for which 
the probability of a configuration X is given by 

Prob(X) = const • ] [ f(x) ■ V 2 (X), X G Con£ N (X), (5.3) 

xex 

where const is the normalizing constant. This process is called the iV-point poly- 
nomial ensemble with the weight function /. 

Proposition 5.1. Let X and f be as above, and f satisfy the assumptions (*), (**). 
Then the N -point polynomial ensemble with the weight function f is a determinan- 
tal point process whose correlation kernel is the normalized Christoffel-Darboux 
kernel (5.2). 

Proof. A standard argument from the Random Matrix Theory, see, e.g., [Me, 
§5.2]. □ 

Remark 5.2. Under a stronger than (*) condition 

£ \x\™~if{x) < 00, 

there exists a monic polynomial pn of degree N, orthogonal to 1, x, . . . , x N ~ x in 
L 2 (X, fix). Then the Christoffel-Darboux kernel can be written as 

1 PN(x)p N -i(y) - pn-ipn(v) 



h N -i x-y 

The value at the diagonal x = y is determined via the L'Hospital rule. 

According to this, the normalized Christoffel-Darboux kernel can be written in 
the form 

K cDr x s _ Vf(x)f(y) p N (x)p N -i(y) - PN-iPN(y) , 54 ^ 
h N -i x-y 

(e) L— ensembles. Let X be an arbitrary discrete space (finite or countably infi- 
nite). We are dealing with the Hilbert space £ 2 (X) = L 2 (X, //), where, as usual, 
\x denotes the counting measure on X. Let Conffi n (X) denote the set of all finite, 
multiplicity free configurations in X (i.e., simply finite subsets). 

Let L be an operator in £ 2 (X) and L(x,y) be its matrix (x,y G X). For X G 
Conffi n (X) we denote by Lx(x, y) the submatrix of L(x, y) of order \X\ whose rows 
and columns are indexed by the points x G X. The determinants det Lx are exactly 
the diagonal minors of the matrix L(x, y). 

We impose on L the following two conditions: 

(*) L is of trace class. 

(**) All finite diagonal minors det Lx are nonnegative. 

28 



Under these assumptions we have 



det L x = det(l + L) < oo. 

XeConf fin (X) 

We agree that detL = 1. Hence, the sum above is always strictly positive. 

Now we form a point process on X living on the finite multiplicity free configu- 
rations X E Conffi n (X) with the probabilities given by 

Prob(X) = (det(l + L))" 1 det L x , X e Conf fin (X). (5.5) 

It is convenient to have a name for the processes obtained in this way; let us call 
them the L-ensembles. 

Proposition 5.3. Let L satisfy the conditions (*) and (**) above. Then the as- 
sociated L-ensemble is a determinantal process with the correlation kernel K = 
L(l + L)-\ 

Proof. See [DVJ, Exercise 5.4.7], [B02, Proposition 2.1], [BOO, Appendix]. □ 

The condition (*) can be slightly relaxed, see [BOO]. The condition (**) holds, 
for instance, when L is Hermitian nonnegative. However, this is by no means 
necessary, see §5(f) below. 

The relation between L and K can also be written in the form 

\-K= (1 + L)- 1 . 

Remark 5.4. Assume that K is a finite-dimensional orthogonal projection oper- 
ator in £ 2 (X) (for instance, K{x,y) = K CD (x,y) as in §5(d)). One can prove that 
there exists a determinantal point process V for which K serves as the correlation 
kernel. V is not an L-ensemble, because 1 — K is not invertible (except K = 0). 
However, V can be approximated by certain L-ensembles. To see this, replace K 
by K e = eK, where < e < 1. The matrices L e = (1 - K e )~ l - 1 satisfy both (*) 
and (**). The process V arises in the limit of the L-ensembles associated with the 
matrices L e as e / 1. One can check that the probabilities 

lim det(l + L e )~ x det(L e ) x , X e Conf fin (X), 

are correctly defined. 

For a special class of matrices L there exists a complex analytic problem the 
solution of which yields the resolvent matrix K. 
We will follow the exposition of [B3] . 

Let X be a discrete locally finite subset of C. We call an operator L acting in 
£ 2 (X) integrable if its matrix has the form 

L(x,x') = l x - x < ' x ^ x > (5.6) 

I 0, x = x'i 
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for some functions fj, gj on X, j = 1, . . . , M, satisfying the relation 

M 

Y,f j (x)g j (x) = 0, xeX. (5.7) 
i=i 

We will assume that fj, gj G £ 2 (X) for all j. 
Set 

/= (/i,---,/m)*, = (91, ■ ■ ■ , 9mY ■ 
Then (5.7) can be rewritten as g t (x)f(x) = 0. We will also assume that the operator 

is a bounded operator in £ 2 (X). For example, this holds for X = Z + c for any 
c G C. 8 Under these assumptions, it is easy to see that L is a bounded operator in 
£ 2 (X). 

Now we introduce the complex analytic object. 

Let wbea map from X to Mat (A;, C), k is a fixed integer. 

We say that a matrix function m : C \ X — > Mat(/c, C) with simple poles at the 
points x G X is a solution of the discrete Riemann-Hilbert problem 9 (X,w) if the 
following conditions are satisfied 

• m(0 is analytic in C \ X, 

• Resm(C) = lim (m(()w(x)) , x G X, 

• m(() — > 7 as C — > 00. 

Here / is the x identity matrix. The matrix is called the jump matrix. 

If the set X is infinite, the last condition must be made more precise. Indeed, 
a function with poles accumulating at infinity cannot have asymptotics at infinity. 
One way to make this condition precise is to require the uniform asymptotics on 
a sequence of expanding contours, for example, on a sequence of circles \(\ = ak, 
Clk — > +00. 

In order to guarantee the uniqueness of solutions of the DRHPs considered below, 
we always assume that there exists a sequence of expanding contours such that the 
distance from these contours to the set X is bounded from zero, and we will require 
a solution m(() to uniformly converge to I on these contours. 

The setting of the DRHP above is very similar to the pure soliton case in the 
inverse scattering method, see [BC], [BDT], [NMPZ, Ch. III]. 

Proposition 5.5 [B3, Proposition 4.3]. Let L be an integrable operator as described 
above such that the operator (1 + L) is invertible, and m(() be a solution of the 
DRHP (X, w) with 

w (x) = -f(x)g(x) t G Mat(M,C). 



8 Indeed, then T is a Toeplitz operator with the symbol Yl — G L°°(S 1 ). 
9 DRHP, for short 
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Then the matrix K = L(l + L) 1 has the form 



K(x,x') = 



where m'(C) = ^ m ^ an d 



G t (x) lim (m'(C) f(x)) , x = x', 



F(x) = lim (m(C) /(a;)) , G(x) = lim ((m^C))" 1 . 



Comments. 1) The continuous analog of this result was originally proved in [IIKS], 
see also [De] and [KBI]. 

2) It can be proved that the solution of the DRHP stated in Proposition 5.5 
exists and is unique, see [B3, (4.9)] for the existence and [B3, Lemma 4.7] for the 
uniqueness. 

3) The requirement of matrix L's vanishing on the diagonal can be substantially 
weakened, see [B3, Remark 4.2]. A statement similar to Proposition 5.5 can be 
proved if the diagonal elements of L are bounded from —1. 

4) Proposition 5.5 holds without the assumptions (*), (**) stated in the beginning 
of this subsection. 

5) If the operator L is bounded, has the form (5.6), but the functions fj and 
gj are not in ^ 2 (X), then it may happen that the operator K = L/(l + L) is 
well-defined while the corresponding DRHP fails to have a solution. 

(f) Special matrices L. Let X be a discrete space with a fixed splitting into the 
union of two disjoint subsets, 

X = X/ U X//. 

The splitting induces an orthogonal decomposition of £ 2 (X), 

e\x) =£ 2 (x I )®f(x n ). 

According to this decomposition we will write operators in £ 2 (X) (or matrices of 
the format X x X) in the block form. For instance, 

[Ln,i Luji J 

where Ljj acts from £ 2 (X/) to £ 2 (X/), Ljjj acts from £ 2 (Xu) to £ 2 (X/), etc. 
We are interested in the matrices L of the following special form: 

(5.9) 

where A is an operator from £ 2 (Xu) to £ 2 (%i) and A* is the adjoint operator. 

For such L, the condition (**) of §5(e) is satisfied, while the condition (*) is 
equivalent to saying that A is of trace class. It can be shown that the construction 
of §5(e) holds even if A is a Hilbert-Schmidt operator, see [BOO, Appendix]. 
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-A* 



A 




Note that the matrices of the form (5.9) are not Hermitian symmetric but J- 
symmetric. I.e., the corresponding operator is Hermitian with respect to the in- 
definite inner product on the space £ 2 (X) defined by the matrix J = J ^ . It 

follows that the matrices K = L(l + L)~ l are J-symmetric, too. This provides a 
class of determinantal processes whose correlation kernels are not Hermitian sym- 
metric. 

Now let us look at an even more special situation. Assume that X is a locally 
finite subset of C such that the operator T defined by (5.8) is bounded. 

Let hi(-), hjj ( • ) be two functions defined on X/ and X// , respectively. We 
assume that hi G £ 2 (Xi), hn G £ 2 (Xn). (The functions hi, hn should not be 
confused with the constants h n attached to orthogonal polynomials, see §5(d).) 

Set 



L = 





-A* 



A 




where A(x, y) = 



h I (x)h II (y) 



x 



y 



(5.10) 



The matrix A is well defined, because x and y range over disjoint subsets X/ and 
X// of X. 

As is explained in [B3, §6], such L is an integrable operator in the sense of §5(e) 
with M = 2. Let us assume that the functions hj and hu are real-valued. Then 
we have L* = —L, and —1 cannot belong to the spectrum of L, that is, (1 + L) is 
invertible. Thus, the DRHP of Proposition 5.5 has a unique solution. 

Let us introduce a special notation for this solution m(Q. We define four mero- 
morphic functions Ri, Si, Rn, Sn by the relation 



m 



run mi2 




" Rl 


-Sn 


777-21 77^22 




-Si 


Rii 



Then the DRHP of Proposition 5.5 for our special L given by (5.10) can be 
restated as follows, see [B3, §6]: 

• matrix elements ran = Ri and 777-21 = —Si are holomorphic in C \ X/j; 

• matrix elements 777-12 = —Sn and 777-22 = Rn are holomorphic in C \ X/; 

• Ri and Si have simple poles at the points of X//, and for x G X// 

Res-Rj(C) = h H (x)Sn(x), 

C=x 

ResSi(C) = hji(x)Rn(x); 

• Rii and Sn have simple poles at the points of X/, and for x G X/ 

Res Ri 1 (C) = h 2 i(x)Si(x), 
Res 5/7(0 = hj(x)Ri(x); 



• Ri, Rii — >■ 1, Si, Sn — * as £ — > 00. 

As before, the last condition is understood as the uniform convergence on a 
sequence of expanding contours such that the distance from these contours to the 
set X is bounded from zero. 
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It can be proved that these conditions imply the relations 

h 2 n (y)Sn(y) _ ^ h H {y)Ru{y) 



Ri(0 = i- E 



Si{0 = _ E (5J1) 

y^-^ii y&Xii 

Ruio = i - E , ^(o = -E^ M § M - ( 5 - 12 ) 

The inverse implication also holds if we know that the functions Ri, Si, Rn, Sn 
have the needed asymptotics as £ — > oo. 

The next statement is a direct corollary of Proposition 5.5. 

Proposition 5.6 [B3, Proposition 6.1]. Let 



m 



Ri —Sii 
—Si Ru 



be a solution of the DRHP stated above, where hi G £ 2 (Xi) and hn G £ 2 (Xn) are 
real-valued. Then (l+L) is invertible, and the matrix of the operator K = Lj (1+L), 
with respect to the splitting X = Xi U Xn, has the form 

K ItII (x,y) = hi(x)hn(y) 
Knj(x,y) = h n {x)hi{y) 
Kii,n(x,y) = hii(x)hn{y) 



x — 


i 

y 


R I (x)Rn(y) - 


- Si(x)Sn(y) 


x - 


- V 


Rn(x)Ri(y) - 


- Sn(x)Si(y) 


x - 


- y 


Rn(x)Sn(y) 


- Sn(x)Rn(y) 


X 


- y 



where the indeterminacy on the diagonal x = y is resolved by the L'Hospital rule: 
Ku(x, x) = hj(x) ((Ri)'(x)Si(x) - (Si)'(x)Ri(x)) , 
Knji(x,x) = hji(x) ((RnY(x)Sii(x) - (Sn)'(x)Rii(x)) . 

(g) Connection between discrete polynomial ensembles and L— ensembles. 

Here we adopt the following assumptions: 

• X = Xi U Xn is a finite or countably infinite subset of R without limit points. 

• The set Xn is finite, \Xn\ = N. 



• hi is a nonnegative function on Xi such that 

Ehj(x) 



+ X' 



(5.13) 



• hi i is a strictly positive function on Xn. 

To these data we associate a function / on X as follows 

hj(x) 



fix) 



n ( x ~ y) 2 ' 

1 

h 2 u{x) n (*-y) 2 
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x G Xi , 
x G Xn . 



(5.14) 



We note that / is nonnegative on X, strictly positive on X/j, and its (2N — 2)nd 
moment is finite, 

J2x 2N - 2 f(x)<oo. 

xeX 

Conversely, given / with such properties, we can define the functions hi and hu 
by inverting (5.14), and then the condition (5.13) will be satisfied. 

Since the function / satisfies the two basic assumptions for a weight function 
stated in §5(d) (the moment of order 2N — 2 is finite, and / is strictly positive on 
an iV-point subset), we can attach to it a discrete polynomial ensemble. 

On the other hand, let us define a matrix L using (5.10). By virtue of (5.13), 
all the columns of A are vectors from £ 2 (X/). Since the total number of columns 
in A is finite, the trace class condition for A holds for trivial reasons. According to 
§5(f), such a matrix L defines a determinantal point process. 

Proposition 5.7. Under the above assumptions, the orthogonal polynomial ensem- 
ble with the weight function f and the L-ensemble associated with the matrix (5.10) 
are connected by the involution A corresponding to Z = %u. 

Proof. We will prove that for any balanced configuration X, the probability of X 
in the the L-ensemble is equal to the probability of X A in the orthogonal polyno- 
mial ensemble. We have to compare two expressions, (5.3) and (5.5), which both 
involve a normalizing constant. Since we know that we are dealing with probability 
measures, we may ignore constant factors. 

Let X be a finite balanced configuration with no multiplicities. Write it as A UB, 
where A = X fl X/ = {ai, . . . , a^}, B = X n %n = {b\, . . . , bd}- In this notation, 
the probability of X in the L-ensemble is equal, up to a constant factor, to 



detL^us = ' ^ et ' 



1 



l<i,j<d 



(5.15) 



For arbitrary finite configurations C = {ci, . . . , c m } and D = {di, . . . , d n } we 
will abbreviate 



V 2 {C)= J] (q- Cj ) 2 , V 2 (C;D) = Hl[(c l -d j y 

l<i<j<m i=lj=l 

By the well-known formula for Cauchy's determinant, 



1 



ai - bj 



V 2 {A)V 2 {B) 
V 2 {A-B) 



the expression (5.15) is equal to 



aeA beB v ' ' 



(5.16) 



On the other hand, X A = A U B, where B = X H \B. The probability of X A 
in the orthogonal polynomial ensemble is equal, up to a constant factor, to 



J] /(*) • V 2 (X A ) = J] f(a) ■ J] fib) .V\AUB). (5.17) 

x£X A aeA beB 
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Let us transform this expression. We have 



Then 



Yl f(b) = const • Yl -tt^t , const = JJ f(x). 

beB bes ^ ' xeXu 



V 2 (A UB)= V 2 {A)V 2 (B)V 2 {A; B) 
V 2 (A)V 2 (B) V 2 (A;B)V 2 (A;B) m „ 



V 2 {A;B) V*(B)V 2 (B;B) 



V\B)V\B)V\B;B) 



+ V 2 (A)V 2 (B) ^AySJ* 
const 



V 2 (A;B) n n (^-2/) 2 

j/€£ii\{6} 



where 



const = V 2 {B)V 2 {B)V 2 {B;B) = V 2 (X n ). 
It follows that (5.17) is equal, up to a constant factor, to 

n(/w n("-yA n(f(») n (*-») 2 ) -^w^f 

aeA V j/exj/ / beB \ yeXn\{b} J ' ' 



By virtue of the connection between / and {hi, fin}, see (5.14), this is equal to 
(5.16). □ 

(h) Connection between two correlation kernels. We keep the assumptions 
of §5(g). In particular, / is related to hj and hjj by (5.14). 

Recall that if we assume that hj G £ 2 (%i) and the operator T (see (5.8)) is 
bounded then the DRHP of §5(f) has a unique solution which defines the mero- 
morphic functions Ri, Si, Rn, Sn. Also, hi G £ 2 (%i) implies that the function 
/ defined through (5.14) has a finite (2iV)th moment, and, hence, we can define 
monic, orthogonal with respect to the weight / polynomials {po,Pi, • • •} at least 
up to the iVth one, see §5(d). Note that the condition hi G £ 2 (3Ci) is stronger than 
(5.13). 

Proposition 5.8. Under the assumptions of§5(g), if hi G £ 2 (%i) and the operator 
T is bounded, then we have 

Rl{0 - n (C-y) ' /(C) " ^-i n (C-y) • ( 5 - 18 ) 



Proof. Denote the right-hand sides of (5.18) by Ri and Si, respectively, and define 
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cf. (5.12). We will show that the matrix 



m = 



Ri ~J?n 
—Si Rh 



solves the DRHP of §5(f). By uniqueness of the solution we will conclude that 
m = m. 

The condition hi G £ 2 (%i) guarantees that the formulas above define meromor- 
phic functions Rn, Sn with needed asymptotics and location of poles. Thus, we 
only need to check the relations involving residues at the poles. The equalities 

Res Ri i (C) = hj^Siix), 



Res S n (C) = h 2 j(x)Ri(x), 



are obviously satisfied. 



The relation Res 57(C) = h 2 II (x)Rn(x) is equivalent to the equality 



Pn-i(x) 2 / hj(y)Si(y) 



hN-i EI (x-y) \ y-x 

y€Xu,y^x \ yizXl 

which can be rewritten as (x G £//) 

-PN-i(x)f(x) Yl (x-t) = -h N -! + PN-i(y)f(y) IJ (y- f )- 



But this is the relation 



(pjv-i(s/), n {y-t)\=h N _ 1 
\ teXn,t=£x I 



which directly follows from the definition of the orthogonal polynomials. 
The relation Res Ri(() = h 2 II (x)Sn{x) is equivalent to the equality 

C=x 



y€Xu,y^x 



which is just the orthogonality relation 



Ff (x — y) y — x 

11 , v yj yeXj y 



(pN(y), J] (y-t))=o. □ 

\ teXn,t^x I 
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Corollary 5.9. Under the assumptions of§5(g), if hj G £ 2 (X) and the operator T 
is bounded, then for any x G Xu we have 



Rn(x) 



Pn-i(x) 



Sn(x) 



Pn(x) 



h N -i hjj{x) 1 1 (x-y)' 



h 2 n (x) n (x-y)' 



(5.21) 



Proof. Follows from the relations 



Res Ri(C) = h 2 II {x)Sjj(x), 



Res 57(C) = h 2 H (x)Rii(x), 



and (5.18). □ 

For the next statement we drop the assumption hi G £ 2 (Xi) and use the weaker 
assumption (5.13) instead. 

Theorem 5.10. Under the assumptions of §5(g), let L be given by (5.10), K = 
L(l + L)~ x , and K CD be the Nth normalized Christoffel-Darboux kernel for the 
weight function f, as defined in §5(d). Introduce the following function on X taking 
values in {±1}; 



where the operation ( ■ ) A is defined by (5.1) with Z = Xu, Z = Xj. 

Before proceeding to the proof let us make a couple of comments. 

Comments. 1) By Proposition 5.3 the kernel K describes the correlation functions 
of the L-ensemble in question. On the other hand, the same correlation functions 
are also expressed in terms of the kernel (K CD ) A , see Proposition 5.7 and §5(c). 
This does not mean that both kernels must coincide, because the correlation kernel 
of a determinantal point process is not defined uniquely, see §5(b). And indeed, we 
see that the kernels turn out to be conjugated by a nontrivial diagonal matrix. Note 
that conjugating by a diagonal matrix is the only possible "generic" transformation, 
because this is the only operation on the "generic" matrix which preserves all 
diagonal minors. In our situation, both kernels are real and possess a symmetry 
property ( J-symmetry) , so that it is not surprising that the diagonal entries of this 
diagonal matrix are equal to ±1. However, the exact values of these ±l's, as given 
in the theorem, are not evident. 

2) The theorem makes it possible to calculate the kernel K = L(l + L) -1 pro- 
vided that we know the orthogonal polynomials with the weight function /. Both 
kernels, K and (K CD ) A , are suitable to describing the correlation functions of the 
L-ensembles. However, from the computations that follow in subsequent sections 
we will see that, for the particular L-ensemble we are interested in, the former 




Then we have 



K(x,y) = e(x)(K^(x,y)e(y), 



(5.22) 
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kernel survives in a scaling limit procedure (see §10) while the latter kernel does 
not. 



Proof of Theorem 5.10. Let us assume that the stronger condition hi G l 2 (3ii) is 
satisfied. Then the normalized Christoffel-Darboux kernel can be written in the 
form (5.4), which makes it possible to express (K CD ) A as follows: 



-CD \ A , , , _ / \ Vf( X )f(y) PN{x)pN-l{y) - PN-l(x)p N (y) 



{K^)? I {x,y) = e{x)s{y) 



h N -i x-y 



(K^)f n (x,y) = e(x)e(y)^ f{x)m P^)Pn-M - PN-i(x)p N (y) 

fiN-i x - y 

(K CD )fi i(x, y) = e(x)e(y) Vf{x)m ^kMz^!^ , 

hn-i x — y 

(K CD )f UI (x,y) 

Vf( x )f(y)PN-i(x)p N (y) -p N (x)p N -i(y) 



8(x - y) + e(x)e(y) 



h N -i x-y 



Here / is the function defined in (5.14). 

Assuming that the operator T of (5.8) is bounded, we see that (5.18), (5.21), 
and Proposition 5.6 directly imply the claim of the theorem everywhere except for 
the diagonal set (x, x) G X x X. But on this diagonal set it is immediately seen 
that both sides of the equality 

K(x,x) = (K CD ) A (x,x) 

represent the probability that the corresponding L-ensemble has a particle at the 
point x, see Comment 1 above. 

Thus, we have verified (5.22) assuming that hi G £ 2 (%i) and the boundedness 
of T. Now we will show how to get rid of these extra conditions. 

Let Hi denote the set of all nonnegative functions hi(x) satisfying (5.13). We 
equip Hi with the weakest topology for which all the evaluations hi i— > hi(x) 
(x G %i) and the sum in (5.13) are continuous. Fix hn and let hi vary over Hi. 
We claim that for any fixed x,y G X, both values K cr> (x,y) and K(x,y) depend 
continuously on hi G Hi. 

By the definition of the topology in Hi, the moments 

°j = J2 xJ f^> j = 0,...,2N-2, 

depend continuously on hi. Let G = [ci+j]^~} Q be the Gram matrix of the vec- 
tors 1, x, ... , x N ~ x G £ 2 (X, ffx). The Christoffel-Darboux kernel can be expressed 
through the moments as follows: 

N-l 

i,j=0 
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(This can be derived from the classical determinantal expression for the orthogonal 
polynomials, see [Er, Vol. 2, 10.3(4)], or by making use of a simple direct argument, 
see, e.g., [B4]. ) Hence 

N-l 

Clearly, this expression is a continuous function of hi. 

On the other hand, again by the definition of the topology, the columns of the 
matrix A(x,y) (see (5.10)), viewed as vectors in £ 2 (X/), are continuous in hi. It 
follows that the operators L and K — L(l + L) _1 are continuous in hi with respect 
to uniform operator topology. Hence K(x,y) is continuous, too. 

Thus, we have proved that both sides of the required equality (5.22) are contin- 
uous in hi G Hi. 

Finally, let H® C Hi be the subset of those functions which have finite support. 
Note that (5.22) holds for any hi G Hj. Indeed, (5.22) does not change if we replace 
%i by supp hi, and for a finite Xj the extra conditions are obviously satisfied. Since 
Hj is dense in Hi, we conclude that (5.22) holds for any hi G Hi. □ 

6. P^) AND 7>( N ) AS DETERMINANTAL POINT PROCESSES 

Recall that in §4 we defined two discrete point processes V {N) and These 
processes live on the lattice %( N > and depend on four parameters (z, z', w, w') G 
D a d m . As was mentioned in §5(c), p( N > and pW are related by the complemen- 
tation principle: = (V^) A , where the special set Z is equal to Xi n . In this 
section we will show that is a discrete polynomial ensemble (as defined in 
§5(d)) and V^ N > is an L-ensemble (as defined in §5(e)). 

Consider the following weight function on the lattice X( N ) : 



JK ' T(z-x+^-)T(z'-x + ^)T(w + x + ^)T(w' + x+^) ' 

- (6.1) 

Here we assume that (z, z' ,w,w') G "P a dm- The expression (6.1) comes from the 
expression for P' N (X \ z, z', w, w'), see §3. Namely, in the notation of §4, we have 

P' N (X\z,z',w,w')=Dim 2 N (X)- 11 f(x). 

xec(x) 

Proposition 6.1. Let (z, z' ,w,w') G T> a d m - The function f(x) is nonnegative on 
%( N > and satisfies the assumptions (*) and (**) of§5(d). 

Proof. The nonnegativity of f(x) follows from Proposition 3.5 and the definition of 
"Dadm, see Definition 3.4. 

The condition (*) of §5(d) says that 

E x 2N - 2 f(x)<oo. 

This follows from the estimate 

f(x) < const -(1 + \ x \)-(z+z'+™+™'+™) 
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and the fact that z + z' + w + w' > — 1 for (z, z', w, w') G T> adm . As for the estimate 
above, it readily follows from the asymptotics of the gamma function, see [013, 
(7-6)]. _ 

Finally, the condition (**) of §5(d) says that f(x) does not vanish at least on N 
distinct points. This follows from the fact that P' N (X \ z, z', w, w') does not vanish 
identically, see Proposition 3.5. □ 

Note that if (z, z') G ^degen then f(x) vanishes for positive large x. Similarly, if 
(w,w') G Zdegen then f(x) vanishes for negative x such that \x\ is large enough. 

Corollary 6.2. Let (z, z' ,w,w') G X? a dm- Then p( N > is a discrete polynomial 
ensemble with the weight function f(x) given by (6.1). That is, for any N '-point 
configuration X = {x±, . . . , xn} 

N 

V (N HX) = const -J] II fa-Xj) 2 - 

i=l l<i<j<N 

Proof. Indeed, by Proposition 6.1, the assumptions of §5(d) are satisfied. The 
formula above is a direct corollary of the definition of Pn(X), see §3, and the 
formula for DiniAr A, see §l(d) . □ 

Now let turn to the process *p( N >. We will apply the formalism of §5(g) for 
X = %( N \ Xi = X ou t, X77 = 3£i n . Recall that §5(g) relies on four assumptions. The 
first three of them hold for any quadruple (z, z', w, w') G "P a dm- As for the fourth 
assumption, it is equivalent to the strict positivity of the weight function f(x) on 
Xin, see §5(g). It may happen that for some admissible quadruples (z, z', w, w') 
this requirement is violated: f(x) vanishes at certain points of Xj n . Specifically, 
this happens whenever (z, z') or (w, w') belongs to ^degen,m with m < 0. For this 
reason we have to impose an additional restriction on the parameters. 

Definition 6.3. Let £>adm denote the subset of D a dm formed by the quadruples 
(z, z', w, w') G "Padm such that neither (z, z') nor (w, w') belongs to ^degen,m with 
m < 0. 

In the rest of the section we assume that (z, z' , w, w') G T>' adrn , so that f(x) > 
for any x G X; n . Note that, in terms of signatures A, this condition means that 
Pn(X I z, z' , w, w') does not vanish at A = (0, . . . , 0). 

Remark 6.4. Recall that in Remark 3.7 we introduced a natural shift on the set 

(z, z' , w, w') 1— > (z + k, z + k,w — k,w' — k), fceZ. 

This shift of the parameters is equivalent to the shift of all configurations of V^ N > 
by k. The definition of "D a dm implies that any quadruple from Dadm \ ^ a dm can De 
moved into T>' adrn by an appropriate shift. Thus, for the study of the process V^ N \ 
the restriction of the admissible quadruples to T>' adm is not essential. This argument 
does not work for the process V^ N \ because the shift above does not preserve the 
splitting X = X out UXi n . However, as will be shown later (see the proof of Theorem 
10.1), the effect of the shift is negligible in the limit transition as — > 00. 
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Let us define functions V'iif^ an d V'ou? on ^in an d X ou t, respectively, by the 
formulas 



- X + Z+ E^^_ x + Z , + N±l^ x + W+ N±l^ x + W , + N±l 



_ T , JV+1 _ r , JV+1 T , N+l , 



JV+1 



^t } (*) = 



((*- 



(6.2) 



r(-a; + ^ + ^±i)r(-x + + ^±i)r(x + w + ^±i)r(x + + ^±1 ) ' 

(6.3) 

where we use the notation 

r(a)r(6)... 



(a) k = F[ ^ k) = a(o + 1) • • • (a + k - 1), T 



r(a) 



a, 6, ... 
c, d, . . . 



r(c)r(d)..." 



Note that 



vffi (*) = <( 



(r(-, + ffi)r(x+ffi)) 2 /(x) 



AT-l 



r (x- 
r(-x + ^i) 



f(x), x<-^ 



The function Vw * s nonnegative and the function ift^ * s strictly positive. Note 
also that both functions are invariant with respect to the substitution 

(z, z') < — > (w, w'), x < — > —x. 

Introduce a matrix of format X x X which in the block form corresponding 
to the splitting X = X ou t U X; n is given by 

o aw 

-(AW)* o 
cf. §5(f), where A^ N ' is a matrix of format X out x X in , 



: (N) 

•^out I— I •^■in 



A^ N \a,b) = 



a — b 



ci G X ut? b G Xi n . 



Proposition 6.5. Let (z, z' ,w,w') G T>' adm . ^ e process J>( N > introduced in $4 
is an L-ensemble with the matrix L,( N > introduced above. That is, for any finite 
configuration X 

(N) 



det(l + LW) 



where denotes the submatrix of of finite format X x X. 

Proof. This is a direct corollary of Propositions 5.7 and 6.1, where for Proposition 
5.7 we take 

X/ = X ou t 5 X/7 = Xi n , hj = V'out^ ^1/ = V'in ^ • 



The relations between the functions Vwt > V'L > an< ^ / above exactly coincide with 
(5.14). □ 
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7. The correlation kernel of the process 



The goal of this section is to compute the normalized Christoffel-Darboux kernel, 
see §5(d), associated with the weight function / on the lattice %( N ) given by (6.1). 
According to Proposition 5.1, this kernel can be taken as a correlation kernel for 
the process J>( N \ We will denote this kernel by K^ N \ 

We will show below that the orthogonal polynomials with the weight / can be 
expressed through the hypergeometric function of type (3,2). This is an analytic 
function in one complex variable u defined inside the unit circle by its Taylor series 



3-^2 



a, 6, c 
e,f 



u 



= E 

k=0 



(a) k (b)k(c)k k 
kl(e) k (f)k ' 



Here a, 6, c, e, / are complex parameters, e, / ^ {0, —1, —2, . . . }. We will only need 
the value of this function at the point u = 1. Then the series above converges if 
3?(e + / — a — b — c) >0. Moreover, the function 



T(e)T(f)T(e + f-a-b-c) 



3 r 2 



a, 6, c 
e, / 



1 



can be analytically continued to an entire function in 5 complex variables a, 6, c, e, f. 10 
Note also that if one of the parameters a, 6, c is a nonpositive integer, say, a G 
{0, —1, —2, . . . }, then the series above has only finitely many nonzero terms. It is 
easy to see that in such a case 



1 



r(e)r(/) 



3-^2 



a, b, c 
e,f 



1 



is an entire function in 6, c, e, /. 

Let us return to the process J>( N \ Set E = z + z' + w + w'. 

Theorem 7.1. For any (z, z' ,w,w') G Dadm? the normalized Christoffel-Darboux 
kernel is given by 

KW(x, y ) = ^- pnWpn-M-pn-iWpnM ,/75)7(i) , (7.i) 



AT-1 



where x,y G £ (N) , 



r(x + «/ + 



JV+l- 



PvU1 iv-„^vr {fj 



pAT-l(z) = ; , r _ 1 2 , ^ 3^2 



—AT, z + w', z' + w' 



r(x + w' - ^ + 1) 



-N + 1, z + w' + 1, z' + w' + 1 
E + 2, x + w' - 



+ 1 



AT-1 



N, E + 1, E + 2 
E + AT + 1, z + w + 1, z + w' + 1, 2/ + w + 1, 2/ + w' + 1 



(7.2) 



10 We could not find a proof of this fact in the literature. We give our own proof in the end of 
the Appendix. 
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and f(x) is given by (6.1). 
Equivalently, 



R{N) ^ y) = 1 PAT^-i^-PAr-iC^Ci/) ^7^) , (7 . 3) 
h>N-i x — y 



where 

—N, z + w' z' + w' 



T{x + W > + N±l 



3 r 2 



E+l,x + w' 



N-l 



(7.4) 



Comment. 1) If E = then the formula for above does not make sense because 
it involves a hypergeometric function with a zero lower index. The formula (7.3) 
gives an explicit continuation of the right-hand side of (7.1) to the set E = 0. 

2) For any x G C, the values pjv_i(x), pjv(x), as well as the constant ftjv-i, are 
analytic functions in (z, z' ,w,w') G V. The value Pn(x) is an analytic function in 
(z, z', w, w') G V \ {E = 0}. If (z, z', w, w') G V then h N - X ^ 0. 

Proof of Theorem 7.1. It is convenient to set 

t = x +^f 1 , u = z + N -1, u' = z' + N-l, v = w, v' = w'. 
Then t ranges over the lattice Z and the function f(x) turns into the function 

9 ^ = T(u + 1 - t)T(u' + 1 - t)T(v + 1 + t)T(v' + 1 + t) ' tGZ ' 
We consider as a weight function on the lattice Z. Note that 
#(t) < const -(1 + | t |)-«-«'-«-«'-2 j | t | ^ 

Indeed, this follows from the estimate of the function / given in the beginning of 
the proof of Proposition 6.1. 

We aim to study monic orthogonal polynomials po = l,pi,p2, • • • corresponding 
to the weight function g(t). In general, there are only finitely many such polynomi- 
als, because, for any (nonintegral) values of the parameters u, u', v, v', the weight 
function g has only finitely many moments. This is a major difference between our 
polynomial ensemble and polynomial ensembles which are usually considered in the 
literature, cf. [NW], [J]. 

The number of existing polynomials with the weight function g(t) depends on 
the number of finite moments of g(t), i.e., on u + u' + v + v'. Specifically, the 
mth polynomial exists if g(t) has finite moments up to the order (2m — 1) (this 
follows, e.g., from the determinantal formula expressing orthogonal polynomials 
through the moments, see [Er, Vol. 2, 10.3(4)]). This condition is satisfied if 
u + u' + v + v' > 2m — 2. Let 



(Pm,Pm) = ^2p 2 m (t)g(t) 



tez 

denote the square of the norm of the mth polynomial. This quantity is well defined 
if g(t) has finite (2m) th moment, which holds if a slightly stronger condition is 
satisfied: u + u' + v + v' > 2m — 1. (Note that it may happen that p m exists but 

{Pm.Pm) = OO.) 
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Proposition 7.2. Set & = u + u' + v + v' and let m = 0, 1, . . . 

// S > 2m - 2 then 



iy + i+j) 



m, u + v' + 1 — m, u' + ?/ + 1 — m 
6 + 2 - 2m, v' + l + t-m 



T(v' + l + t-m) 
If 6 > 2m - 1 tfien 

(Pm,Pm) = ^2p 2 m (t)g(t) = 

m + 1, & + 1 - 2m, 6 + 2 - 2m 
6 — m + 2, w + -u + 1 — m, w + f ' + 1 — m, u' + t> + 1 — m, u' + v' + 1 — m 

We will give the proof of Proposition 7.2 at the end of this section. Now we 
proceed with the proof of Theorem 7.1. 

Note that the condition ~E = z + z' + w + w' > —1 entering the definition of 
D a( j m is equivalent to the condition & = u + u' + v + v'> 2N — 3. This implies 
the existence of the iVth Christoffel-Darboux kernel associated with the weight 
function g(t), 



N-1 



Hence, the iVth Christoffel-Darboux kernel associated with the weight function / 
has the form 



n \Pmi Pm) 

m=0 

By the general definition of the normalized Christoffel-Darboux kernel (see (5.2)), 

K( N \x,y) = S( N \x,y)^f(x)f(y). 

On the other hand, let t[ N \x, y) denote the right-hand side of (7.1) with the term 
\J f{x)f{y) removed. Similarly, let T 2 (x, y) denote the right-hand side of (7.3) 
with the term a/ f(x)f(y) removed. It suffices to prove that 

S^ N \x,y) = Ti N \x,y) = 4 N \x,y). 

Lemma 7.3. For any x,y £ X^ N \ the kernel S( N \x,y) viewed as a function in 
(z, z', w, w'), can be extended to a holomorphic function on the domain V . 

Recall that the domain T>$ was introduced in §3. 

Proof. Indeed, the claim of the lemma holds for p m (x — jV ~ 1 ), p m (y — jV ~ 1 ), and 
(PmjPm)" 1 ) where m = 0, . . . , N — 1. This can be verified using explicit formulas 
of Proposition 7.2. □ 

Let us prove first that S^ N \x,y) = T^ N \x,y) under the additional restriction 
E > 0. Then & > 2N — 2, so that the iVth polynomial p^ exists. Hence, the kernel 
S( N ^ can be written in the form (see Remark 5.2) 

s w (x,y) 

1 PN (x - ^)p N -i(y - V) - Pn-i(x - ^)p N (y - gg) 

(pn-i,Pn-i) x-y 

44 



By explicit formulas of Proposition 7.2, 
p N (x - ^f 1 ) = p N (x), p N -i(x - ^f 1 ) = p N -i(x), (pn-i,Pn-i) = h N -!. 

This implies the desired equality when E > 0. 

Next, observe that the expression T[ N \x,y) is well defined when (z, z', w, w') 
ranges over the domain Vq \{E = 0} and is a holomorphic function on this domain. 
Together with Lemma 7.3 this proves the equality (x, y) = T^ N ^ (x, y) provided 
that (z, z' ,w,w') E Vq \ {E = 0}. As a consequence, we get (7.1) under the 
restriction E ^ 0. 

Finally, the series representation for 3F2 easily implies the identity 



3-^2 



a, 6, c 
e,f 



— 3F2 



a, 6, c 



e + 1, / 



1 



e(e + l)/ 3 2 



a + 1, 6+1, c+1 
e + 2,/+l 



1 



Applying this identity to pN we see that Pn(x) = Pn(x) + const pjv-i(^)- It follows 
that T± N \x,y) = T 2 (x,y) provided that Pn(x) makes sense. Thus, we see that 
the singularity in T± (x, y) on the hyperplane E = is removable. Specifically, this 
singularity is explicitly removed by means of the equality T^ N \x, y) = T 2 (A °(z, y) 
(equivalently, by means of (7.3)). This completes the proof of Theorem 7.1. □ 

Proof of Proposition 7.2. We apply the general formalism explained in [NSU, Chap- 
ter 2] . Consider the following difference equation on the lattice Z: 

a(t) AVy(t) + r(t) A y(t) + 7 y(t) = 0, (7.5') 

where 

a(t) = -(t + v)(t + v'), 

r(t) = et + vv' -uu', (7.5") 
Vy(t) = y(t) - y(t - 1), Ay(t) = y(t + 1) - y(t). 

According to [NSU] this equation is of hypergeo metric type, that is, a{t) is a poly- 
nomial of degree 2, r(t) is a polynomial of degree 1, and 7 is a constant. The crucial 
fact is that this equation can be rewritten in the selfadjoint form with the weight 
function g(t): 

(Aoago V)y + 7 gy = 0, 
which easily follows from the relation 

g(t) a(t+l) 



g(t + l) a(t)+r(t) 

(Note that in [NSU] the weight function is denoted by p and the spectral parameter 
7 is denoted by A.) 

We will seek p m as a monic polynomial of degree m, which satisfies the difference 
equation above with an appropriate value of 7. According to [NSU], this value must 
be equal to 

, m(m - 1) „ 
7 = 7m = -mr a , 
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where the derivatives r' and a" are constants, because r has degree 1 and a has 
degree 2. Further, the polynomial in question exists and is unique provided that 



k(k — 1) 

[ik ■= lm + kr' + — VVO, fc=0,...,m-l. 
In our case r' = 6, a" = — 2, so that 

7m = m(m - 1 - 6) 
and the nonvanishing condition /j,k ^ turns into 

6^m + fc-l, fe = 0, ...,m— 1, 

which is certainly satisfied if & > 2m — 2. 

Our additional condition E > is equivalent to & > 2N — 2. It follows that the 
desired polynomial solutions p m certainly exist for m < N. 

Following the notation of [NSU] set 

m — 1 



A mm = ml JI( T '+™±*-V), 

m 

9m(t)=9(t)l[a(t + l), 

1=1 



Then, according to [NSU], 



p ( x ) - (~ 1 ) m?Tl! (-m) k g m (x -m + k) 
_ (-ir(ml) 2 S m 

\Pmi Pm) , 



Applying this recipe in our concrete case we get 

r(6-m + 2) 



9m(t) 



r(6-2m + 2) ' 

(_l)m 



r(« + 1 - m - t)r(u' + i - m - t)r(^ + 1 + *)r(v' + 1 + 1) 

and, by Dougall's formula [Er, Vol.1, 1.4(1)], 



= (-i) m r 



tei 

6 + 1 - 2m 

u + v + 1 — m, it' + i/ + 1 — m, i/ + i; + 1 — m , it + i/ + 1 — m 
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This implies the expression for (p m ,Pm) given in Proposition 7.2. 
Further, after simple transformations we get 



Pm(t) =r 



= r 



6 

Applying the transformation 



6 - 2m + 2, v + 1 + £, v' + 1 + * 
© — m + 2, u + 1 — m + t, + 1 — m + t 

E (-m) k (t - u) k (t - u') k 
, n (t + u + l -m)fc(t + u' + l -m) k k\ 

k=0 

&-2m + 2, v + l + t, v' + l+ t 
m + 2, i> + 1 — m + t, v' + 1 — m + t 



-m, t — u, t — v! 



t + v 



+ 1 — m, t + v' + 1 — m 



3 r 2 



a, 6, c 
e,f 



e, e + f — a — b — c 
e + f — b — c, e — a 



3 r 2 



a, f -b, f - c 
e + f-b-c,f 



(7.6) 



see Appendix, to the last expression we arrive at the desired formula for p m (t) . □ 

The orthogonal polynomials {p m } were discovered by R. Askey [As]. Later they 
were independently found by Peter A. Lesky, see [Lesl], [Les2]. We are grateful to 
Tom Koornwinder for informing us about Lesky's work, and to Peter A. Lesky for 
sending us his preprint [Les2]. 



8. The correlation kernel of the process 

In this section we study the process V^ N ' on the lattice %( N ' introduced in §4. In 
56 we showed that P (Af) is an L-ensemble, hence, it is a determinantal point process. 
Denote by the X (iV) x X (JV) matrix defined by = L^fl+lW)" 1 , where 
was defined in §6. By Proposition 5.3, is a correlation kernel for p( N \ 

Our goal in this section is to provide analytic expressions for suitable for a 

future limit transition as N — > oo. 

Our analysis is based on the general results of §5(f)-(h), where we takeX = X (Ar) , 
Xi = X ou t, %u = Xin- To compute the kernel K^ N > we use the method of §5 (f ) . 
Proposition 5.6 expresses the kernel in terms of four functions Ri, Rn, Si, Sn, 
which solve a discrete Riemann-Hilbert problem. In our concrete situation we will 

(TV) 



redenote these functions by -Rout ) -^in^ 



'-'out 5 °" 



In order to apply this method we need to impose an additional restriction on 
the parameters. Specifically, we require that (z, z', w, w') G "Cadm H {E > 0}, where 
T>' adm was defined in Definition 6.2 and £ = z + z' + w + w'. Later we show that 
the final expression for remains valid without the restriction E > 0. 

Thanks to Proposition 5.8, the functions R^ and S^ are immediately ex- 
pressed through the orthogonal polynomials pN, pjv-i evaluated in §7. The re- 



maining two functions and S^ ' are uniquely determined by their relations 



JN) 



to r£$ and S out . 



,(iV) 



see (5.12) and (8.3) below. We provide certain explicit expres- 
sions for , S\n ^ an d check that they satisfy the needed relations. About the 
derivation of these expressions, see Remark 8.9 below. 
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To remove the restriction E > we use Theorem 5.10 and Lemma 7.3. These 
results show that the kernel K^ N > divided by a simple factor, admits a holomorphic 
continuation (as a function of the parameters) to a certain domain V D T>' adin . 

Now we proceed to the realization of of the plan described above. 

We assume that (z, z', w, w') G T>' adm - As was mentioned in §6, this guarantees 
that the weight function f(x) does not vanish on X- in . Next, we temporarily assume 
that E > 0. This ensures that the function hi belongs to £ 2 (X in ) as required in 
§5(h). ' 

Following Proposition 5.8 and using Theorem 7.1, we define 2 meromorphic func- 



tions and S y out 



,(iV) 



on the complex plane with poles in Xi n as follows: 
p N (x) 



(x 



N-l 
2 

X — 



X + w + 



x + ^r 1 , x + w' 



2 

N-l 



3 r 2 



-N, z + w', z' + w' 



(8.1) 



and 



S Q ut( x ) — 



pN-l(x) 



h 



N-l 



(x-^)-..(x + ^) 



N-l 



h-N-i 



x — 



N-l 



X + W ' + N±l 



x+ E±l jX + w > 



2 

3-^2 



-N + 1, z + w' + 1, z' + w' + 1 
E + 2, x + w' - ^ 



1 

(812) 



where h^-i was defined in (7.2). Observe that 



r£2{x) = 1 + 0{x- 1 ), SC^Oix- 1 ), x 



(N) 



oo. 



Note that the right-hand side of (8.1) makes sense for any x G C \ X- in and 
(z, z' , w, w') G V \ {E = 0}, and the right-hand side of (8.2) makes sense for any 
x G C \ Xi n and (z, z' ', w, w') G Vq. 



Using (5.12) as a prompt, we set 



y&X ou t 



X 



!" E 



x 



(8-3) 

where the functions V'ou? an d V'in were introduced in §6. 

Note that Vw?^) — const (1 + |a;|) _E (this follows from the estimate of the 
weight function /, see §6). Since, by assumption, E > 0, it follows that the series 
(8.3) converge and define meromorphic functions with poles in X ou t- 

Proposition 8.1. We have 



R 



(N) 



smnz 

(x) = r 



TV 



z' — z, z + w + 1, z + w' + 1 
E + l 



x r 



X + 



N+l 



_ x + z ' + N+l' 



-X + 



N+l 



iV + l + E 



x + w + ^±1 , N + 1 + z + w' 



X 3^2 



2 , ~ i ^ , 2 
Z + w' + 1, — z' — W, —X + -2 + ^2^- 

z-z' + l, N + l + z + w' 



1 



(8.4) 



— { similar expression with z and z interchanged }, 
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,(N) 



smnz 

(x) = r 



TV 



z' — z, £ 
z' + w, z' + w' 



x r 



X + 



AT+1 



^±±,iV + l 



2 " i X -\- 

-x + z' + ^±±, x + w + ^P-, N + l + z + w' 



w + 1, z + u/, —x + 2 + 



jV+1 



(8.5) 



z-z' + l, iV + l + z + u/ 
— { similar expression with z and z interchanged }. 

Singularities. Using the structure of singularities of a general 3F2 function, see 
the beginning of §7, it is easy to verify that the formulas above make sense for 
x eC\ X out and (z, z', w, w') eV \ ({£ = 0} U {z - z' e Z}) . {E = 0} is indeed 

a singular set for . The singularities {z — z' G Z}, however, are removable, as 
long as we are in T>q \ {E = 0}. For E > this follows from the definitions (8.3). 

Proof. Since both parts of (8.4) and (8.5) are analytic in the parameters, it is 
enough to give a proof for nonintegral values of z,z',w,w', such that z — z' ^ Z, 
E £ Z. 

We start with rewriting the expressions above in a somewhat more suitable form. 
Introduce a meromorphic function 

F(x) = - 

sm(Tv(z' — z)) sm(Tv(z + z' + w + w')) 

sin(7r(,2 + w)) sin(7r(z + w')) sin(Tvz') sin(7r(z' + w)) sin^^' + w')) sin(7r^) 



x 



sm{7i{-x + z' + ^)) 



sm(n(-x + z + ^±i)) 



It is not difficult to see that if x £ 3£, that is, if x 



N-l 



Z, then F(x) = 
(-1)"-^. 

Let us also introduce a more detailed notation h(N — 1, z, z', w, w') for the con- 
stant hN-i defined in (7.2). 

Lemma 8.2. The right-hand side of (8.4) can be written in the form 



np _| I 1 rp y 



AT-1 



_ N-l _ , JV+1 



3-^2 



A/", — 2; — w', —2 — 
S, x z -\- — ^— 



1 



+r 



-x + z + -x + z' + X + w + x + w' - ^ 



X 



Fix] 



h(N - l,z, z',w,w') 



3-^2 



—AT + 1, z + w' + 1, 2/ + w' + 1 
E + 2, x + w' - ^ 



and the right-hand side of (8.5) can be written in the form 



]i[N,z-^,z'-±w-^,w'-^)\ 



1 N+l _ _ N-l 
iXj 1 2 1 «*•' & 2 

_ iV-1 _ , AT+3 



X 3 ^2 



AT + 1, -z - + 1, -z - w + 1 



+r 



— E + 2, x — z + 



N+3 
2 

AT+1 



- x + z + N±l f - x + z > + N±l f x + w + N±l 5 x + w > _ N-l 



xF{x) 3F2 



2 , \ ™ \ 2 , 

-N, z + w' z' + w' 



X,x + w'-^ 
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1 



(8.6) 



(8.7) 



Singularities. It looks like (8.6) may have more singular points that the right-hand 
side of (8.4). For example, the first summand in (8.6) has poles at the points, where 



x 



u 



N- 



i e{o,-i,...}, 



u = z or z 



and same is true about the second summand. The fact that these poles cancel out 
is not obvious. Similar cancellations happen in (8.7). 

The proof of the lemma can be found in the Appendix. It is rather tedious and 
is based on the known transformation formulas for 3F2 with the unit argument. 
Our next step is to prove the following statement. 

Lemma 8.3. The only singularities of the right-hand of (8.4) regarded as a func- 
tion in x G C, are simple poles at the points o/X ou t- The residue of the right-hand 
side of (8.4) at any point x G X ou t equals tpiut ( x )^out ( x ) ■ Similarly, the only sin- 
gularities of the right-hand of (8.5) regarded as a function in x G C, are simple 
poles at the points of 3C out . The residue of the right-hand side of (8.5) at any point 
x G X out equals ^^(x)R^(x). 

Proof. The location of poles part follows from the general structure of singularities 
of 3F2, see the beginning of §7. 

To evaluate the residue of the right-hand side of (8.4) we will use the formula 
(8.6). We easily see that first summand of (8.6) takes finite values on X out . As 
for the second summand of (8.6), the 3 F 2 is a terminating series and it has no 
singularities in X out . Furthermore, F(x) = (— l) x ~ for x G X, in particular, for 
x G Xout- It remains to examine the residue of T(x + ^^-)T(—x + ^f-). 



Assume that x G X ou t and x > 



N-l 



Then 



r,, m > Z / 



(-1) 5 



r(x- 



N-l 



Thus, the residue of (8.6) at x is equal to 



x + 



N+l 



N+l 
2 

N+l 



N+l 



JV-3 



1 



X 



h(N - l,z,z',w,w') 



3-^2 



-N + 1, z + w' + 1, z' + w' + 1 
E + 2, x + w' - ^ 



By a direct comparison we see that this is equal to V'out ( x )^ouV( a; ) 



Similarly, if x G X ou t and x < 



N-l 

2 ' 



we have 



Re S r( W +^±i) 



(-1)- 



JV + 1 



n-x-^y 



and the residue of (8.6) at x equals 



-x + 



N+l 



X 2 7 X I Z I 2 ' *^ I ^ I 2 ^ X I XI) I 2 ^ X I U) 2 



X 



(-1) 



N 



h(N — 1, z, z', w, w') 



3-^2 



-N + 1, z + w' + 1, z' + w' + 1 
E + 2, x + w' - ^ 
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which is again equal to V , o^(^)^ou?( a 0- 

Thus, we have proved the first statement of the lemma. The proof of the second 
one is very similar. □ 

Lemma 8.3 shows that the right-hand sides of (8.3) and of (8.4), (8.5) have the 
same singularity structure. Clearly, the sums in the right-hand sides of (8.3) decay 
as x — > oo and x keeps finite distance from the points of X ou t- We aim to prove 
that the right-hand sides of (8.4) and (8.5) have a similar property. Using Lemma 
8.2, we may consider the expressions (8.6) and (8.7). 

Consider the formula (8.6). We first assume that $lx > 0. Let us examine 
the first summand. The gamma-factors form a rational function which tends to 1 
uniformly in arg(x) as \x\ — > oo. 

In order to handle the asymptotics of the 3F2 factor, let us apply the formula 
(7.6) to it with 

N+l 



a = 



We get 



3-^2 



N, 



w, 



—z - 
E, x 



b = 



w 



c = N, e = x — z + 



/ = -£• 



w 



z + 



■z — w 

N+l 



X — Z + 



N+l 



x — z — z — w 



x — z — 

N-l 



N-l 



X + W + 



N+l 



X3F2 



—z — w, —z — w, — E — N 

v-\ / N—l 

— E, x — z — z — w — +++- 



Using the asymptotic relation 



(8.8) 



w 



hich is uniform in arg(x) G (— tv + e, % — e) for any e > 0, we see that the gamma- 



factors tend to 1 as | x \ — > 00. 

As for the 3F2, we see that the sum of the lower parameters minus the sum of 
the upper parameters is equal to x + w + Then if > and 3iw + > 0, 

the defining series for this 3-F2 converges uniformly in x, provided that the distance 
from x to the lattice Z + z + z' + w + is bounded from zero. One proof of this 
fact can be obtained by applying the general estimate (see [Er, Vol. 1, 1.9(8)]) 

a > 0, UP, 3?7 > const > 0, 

T(a + 7) 

ci, c 2 > are some fixed constants, to the terms of the series. 

Thus, we can pass to the limit \x\ — > 00 term- wise, and since all terms of the series 
starting from the second one converge to zero uniformly in arg(x), we conclude, that 



»_„ T(a + /3) 
C\OT 7 < -f < c 2 u p 7 , 



00, converges to 1 uniformly in x such that 

- is bounded 



the first summand of (8.6), cLS X I 

5to > and the distance from x to the lattice Z + z + z' + w + — 
from zero. 

Let us proceed to the second summand of (8.6). Now the 3F2 part is a rational 
function which tends to 1 as \x\ — * 00 uniformly in arg(x). The remaining part - 
the product of gamma-factors and F(x) — can be written in the form 

N+l 



const 1 T 



+ const2 r 



X + 

-x + z' + 



X 



—x + z + 



2 ' 
N+l 
2 ' 
N+l 
2 ' 
N+l 



-X + 



N+l 



X + W + 



N+l 



N-l 



N-3 



-X + 



N+l 



X + W + 
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, x - 

N+l 



X + w — 

„/ _ JV-1 

* 2 

x + w'-^ 



(8.9) 



where for F(x) we used the formulas (u = z or z') 

1 _ F(-x + u + n±±)T(x-u- 

sin(7r(-x + u + -^p)) n 

Further, using the relations (u = z or z') 

T(-x+^) __ sin(7r(-x + w+ H±l))T(x-u- 
T(-x + u+^p) ~ sin(7r(-x + ^i))r(x- ^i) ' 

observing that the ratio of sines is bounded as long as x is bounded from the lattice 
X = Z + and employing (8.8), we see that the absolute value of (8.9) is 

bounded by a constant times |x| _E_1 , and the bound is uniform in x such that 
arg(x) is bounded from ±tt and x is bounded from X. Since E + 1 > 0, the second 
term of (8.6) tends to zero. 

We conclude that, under the condition $tw + > 0, the expression (8.6) 

converges to 1 as \x\ — > oo uniformly in x such that Jto > and the distance from 
x to the lattices Z + and Z, + z + z' + w+ jV ~ 1 is bounded from zero. 

To extend this estimate to the domain 9fcc < we use the following 

Lemma 8.4. The expression (8.6) is invariant with respect to the following change 
of variable and parameters: 

x I— > —x, (z, z') < — ► (m/, w). 

We give a proof of this lemma in the Appendix. Similarly to the proof of Lemma 
8.2, it is rather technical and is based on certain transformation formulas for 3F2. 

Lemma 8.4 immediately implies that, under the conditions $lw + > 0, 

$tz' + > 0, the expression (8.6) converges to 1 as \x\ ^00 uniformly in x 
such that the distance from x to the lattices Z + ^f^, Z + z + z' + w + ^f^, 
Z + z' + w + w' + is bounded from zero. 

This statement together with Lemma 8.3 shows that the right-hand side of the 
second formula of (8.3) and the right-hand side of (8.4) have the same singularities 
and asymptotics at infinity. Thus, they must be equal. This proves (8.4) under 
the additional conditions ^tw + > 0, + ^ > 0. Since both sides of (8.4) 
depend on the parameters z, z' , w, w' analytically, the additional conditions may be 
removed. The proof of (8.4) is complete. 

The proof of (8.5) is very similar. It is based on the following technical statement, 
which will also be addressed in the Appendix. 

Lemma 8.5. The expression (8.7) is skew-invariant with respect to 

xh-x, (z,z')< — >(w',w). □ 

The proof of Proposition 8.1 is now complete. □ 

Remark 8.6. Having proved the formulas (8.6), (8.7), it is easy to verify (5.21) 
directly. Indeed, if x e £i n then the first summands of (8.6) and (8.7) vanish thanks 
to T(x + ^p-)/T(x — ^f^), while in the second ones we have F(x) = (— l) x ~ . 
A direct comparison of the resulting expressions with (8.1) and (8.2) yields (5.21). 
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Theorem 8.7. Let (z,z',w,w') G T>' adrn (see Definition (6.3)), be the corre- 

sponding point process defined in and K^ N "> = L^ N \l + Z/ JV )) -1 , where is 
as in §6. 

Then the kernel K^ N \x,y), represented in the block form corresponding to the 
splitting X = X out U £ in , is equal to 



if 1 ; 

- fv out, 



< n (*,y) = VC v V)V>r(y) 





X - 


- V 






1 -"-out 




-S {N) (x 




><») 




X - 


- y 






1 in 


'(^^(y)- 










x - 








R {N) 


(rf(y)- 




i?. (JV) 

in 


(</) 




x — 






> 


^in 


are given by 


(8.1), (J 


1.2), 


(8.4) 



the functions tp^t > Vnf^ are 9^ ven by (6.2), (6.3), and the indeterminacy on the 
diagonal x = y is resolved by the L'Hospital rule: 



^out^out^ x ) ~ tPout ( X ) ((-^out) ( x )^out( x ) (Sout) ( x )Rout( x ) 



Singularities. We know that the functions i?out an d ^hf^ are singular when E = 0, 
see the beginning of the section. However, as will be clear from the proof, the value 
K( N ' {x, y) is a well-defined continuous function on the whole D a( j m including the 
set {E = 0}, for any x, y G %( N \ 

Proof. Under the additional restriction E > the statement of the theorem follows 
from the above results. Indeed, the functions R^i \ ^out > Rin \ defined by 

(8.1), (8.2), (8.3) solve the discrete Riemann-Hilbert problem associated with L^ N \ 
see (5.18) and (5.12). Proposition 5.6 explains how the kernel is written in 

terms of these functions. Proposition 8.1 provides explicit expressions (8.4), (8.5) 
for -Rjn , and this completes the proof when E > 0. 

Now we want to get rid of this restriction. 

Introduce a function vjrW : %( N ) — > C by 



x G X 



in • 



We have 



*^(x) = /(x).|« X -^W a ' . XGX ° Ut ' (8.10) 
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where f(x) was defined in (6.1). Note that tfWfc) is entire in (z, z' ', w, w') for any 
Set 

^ o = {(*, w, w') e A) U ™, w' £ -1, -2, . . . }. 

Recall that Do contains £> a dm and note that D fl "D a dm = ^ 1dm- m particular, V 
is a domain in C 4 containing T>' adm . 

Lemma 8.8. Let (z,z',w,w 

') e V 'a,dm- The kernel K^ N \x,y) can be written in 

the form 

K( N \x,y) = y/y( N )(x)yW(y)k( N \x,y), (8.11) 

o 

where K^ N \x,y) can be extended (as a function in (z, z',w,w')) to a holomorphic 
function on the domain V . 

Proof of the lemma. Recall (see §7) that 

K( N \x,y) = S^ N \x,y)^f(x)f(y), 

where S^ N ' is the (ordinary, not normalized) Christoffel-Darboux kernel, which 
admits a holomorphic continuation to the domain V (see Lemma 7.3). Theorem 
5.10 provides a connection between the kernels and K^ N \ Specifically, (5.22) 

reads 

K^(x,y) = e(x)(KW) A (x,v)e(y), 
where the symbol (-) A i s explained in §5(c). Since e( • ) does not depend on 
(z, z' ,w,w'), it suffices to prove that the kernel (f^) A (x, y) can be represented 
as the product of ^~^>^ N \x)^>^ N \y) and a holomorphic function on V . 

By (8.10), ty( N \x) differs from f(x) by a positive factor not depending on 
(z, z',w,w'). Hence, it is enough to prove that (K^ N >) A (x, y) can be written as 
the product of a/ f(x)f(y) and a holomorphic function on V . 

By (5.1) we have 

(i(T(^) = f /(l) ^" S(N)(l,l) )' I = !/eIm ' 

I ±V f(x)f(y)S ( ^ N ' ) (x,y), otherwise, 

where the sign '±' does not depend on (z, z', w, w'). From the definition of f(x) it 
follows that for x E Xi n , j^y admits a holomorphic extension (as a function in the 

parameters) to T>' Q . Since S( N ^(x,y) is holomorphic on Do? the lemma follows. □ 

Now we complete the proof of Theorem 8.7. Since we have already proved its 
claim when E > 0, we see that under this restriction 

^out.out^) V) — x - y ' 



x y 

^h^y) = * iN) (x)* {N) (y) 



RS\z)sL N \y)-sL N \x)Ri N \y) 



x-y 
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Since the functions -R out , S£ ut , -R in , Sf n , and vjrW are all holomorphic on 
£>o \ = 0}, these formulas hold on V , and the singularity at E = is removable. 
As V D D^ dm , the proof of Theorem 8.7 is complete. □ 

Remark 8.9. The reader might have noticed that the proof of Proposition 8.1 
that we gave above is a verification that our formulas give the right answer rather 
than a derivation of these formulas. Unfortunately, at this point we cannot suggest 
any derivation procedure for which we would be able to prove that it produces the 
formulas we want. However, we can explain how we obtained the answer. 

Recall that in our treatment of the orthogonal polynomials in §7 the crucial role 
was played by the difference equation (7.5). Corollary 5.9 shows that if we know a 
difference equation for the orthogonal polynomials then we can write down differ- 
ence equations for the values of R[^ and on ^ ne lattice X; n . At this point we 
make the assumption that the meromorphic functions R^ (0 and S^\() satisfy 
the same difference equations. A priori, it is not clear at all why this should be the 
case. However, the general philosophy of the Riemann-Hilbert problem suggests 
that Rl"\0 and s£f°(C) should satisfy some difference equations. So we proceed 
and find meromorphic solutions of the difference equation which we got from the 
lattice. This can be done using general methods of solving difference equations 
with polynomial coefficients, see [MT, Chapter XV]. We also want our solutions to 
be holomorphic in C \ X ou t and to have a fixed asymptotics at infinity, and this 
leads (through heavy computations!) to the formulas of Lemma 8.2. Unfortunately, 
these formulas are not suitable for the limit transition N — > oo. So we had to play 
around with the transformation formulas for 3F2 to get the convenient formulas of 
Proposition 8.1. 

9. The spectral measures and continuous point processes 
Define a continuous phase space 

X = M\{±i} 

and divide it into two parts 



X Xj n U X ou t ? 
£in = (-§,§), ^out = (-OO, -|) U (±,+Oo). 



As in §4, by Conf(X) we denote the space of configurations in X. We define a 
map t : Q — > Conf(X) by 

^ C(u) = {at + §} U {\ - (3+} U {-aj - \} U {-§ + (9.1) 

where we omit possible 0's in ct + , (3 + , a~ , (3~ , and also omit possible l's in (3 + or 
P~. 

Proposition 9.1. The map (9.1) is a well-defined Borel map. 

Proof. We have to prove that, for any compact set Ac I, the function Na(l(u))) 
takes finite values and is a Borel function, see the definitions in the beginning of 

55 



§4. Without loss of generality, we may assume that A is a closed interval, entirely 
contained in (—3, \), (§, +00) or (—00, — |). 
Assume that A = [x, y] C (— |, ^). Then 

A/^M) = Card{z I § - y < /?+ < \ - x) + Card{j | \ + x < Pj < \ + y} 

00 00 

= Yl 1 [l/2-y,l/2-x](A + ) + 1 [l/2+x,l/2+y](/37), 
i=l j=l 

where 1[ ] stands for the indicator of an interval. Since Yl^t — S P J — $~ ■> 
and |+x>0, ^ — y > 0, the sums of the indicator functions above are actually 
finite. Clearly, these are Borel functions in u. 

When A is contained in (^, +00) or (—00, |), the argument is the same. □ 

Let x be a character of the group U (00) and let P be its spectral measure. By 
Proposition 9.1, the pushforward i{P) of P is a well-defined probability measure 
on the space Conf(X). We view it as a point process and denote it by V. Our 
purpose is to describe the spectral measure P (for concrete characters x) m terms 
of the process V. The map 1 glues some points ui together, and it is natural to ask 
whether we lose any information about P by passing to V . We discuss this issue 
at the end of the section. 

Let {Pn} be the coherent system corresponding to x- Recall that in §2(c) we have 
associated with each Pn a probability measure P_ N on O, which is the pushforward 
of Pn under an embedding of G1V into Q. According to Theorem 2.2, the measures 
P N weakly converge to P as N — > 00. Let us form the probability measures 

which are point processes on X. Theorem 2.2 suggests the idea that the process V 
should be a limit of the processes as N — > 00. For instance, the correlation 

measures of V should be limits of the respective correlation measures of 2 ' s - 
Then this would give us a possibility to find the correlation measures of V through 
the limit transition in the correlation measures of the processes V_( N ^ as N — > 00. 
The goal of this section is to provide a general justification of such a limit transition. 

First of all, let us give a slightly different (but equivalent) definition of the 
processes V {N) . Recall that in §4, we have associated to each Pn a point process 
V {N) on the lattice X {N) . Consider the map 

^Ufl^xWcl, x ^ly- (9 - 2) 

Then the process can be identified with the pushforward of the process 

under this map of the phase spaces. Hence, denoting by and pj^' the /cth corre- 
lation measure of and V^ N \ respectively, we see that is the pushforward 
of p^ under the map (9.2). 

Let us assume that V (N) is a determinantal process with a kernel K^ N >{x,y) 
on %( N } x %( N } (which actually holds in our concrete situation). Then P} N > is 
a determinantal process, too. It is convenient to take as the reference measure 
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on the lattice X} N ^ not the counting measure but the measure such that 

fj,( N )({x}) = jj for any x G X} N >. The reason is that, as N — > oo, the measures 
/j,( N ^ approach the Lebesgue measure on X. Taking account of the factor we see 
that the kernel 

KW(x,y) = N-KW(Nx,Ny), x,yeX (N) . (9.3) 

is a correlation kernel for V^ N ^ . 

We need one more piece of notation: given x G X, let x^v be the node of the 
lattice %( N > which is closest to Nx (any of two if Nx fits exactly at the middle 
between two nodes). 

The main result of this section is as follows. 

Theorem 9.2. Let x be a character of U(oo) and let P, {Pn}, X, V, be 
as above. Assume that each is a determinantal process X^ with a correla- 

tion kernel K^ N \x,y) whose restriction both to x x\^ and X^ x X^ is 

Hermitian symmetric. Further, assume that 

lim N • K( N \x N , y N ) = K(x, y), x.yeX, 
uniformly on compact subsets of Xx X, where K(x,y) is a continuous function on 

Then V is a determinantal point process and K(x, y) is its correlation kernel. 

The proof will be given after a preparation work. First, we review a few necessary 
definitions and facts from [013]. 

A path in the Gelfand-Tsetlin graph GT is an infinite sequence t = (t±, t^-, ■ ■ ■ ) 

such that t N G G1V and £/v -< *at+i for any iV = 1,2, The set of the paths 

will be denoted by T. 

Consider the natural embedding Tc [] GTat. We equip Yl GTn with the prod- 

N N 

uct topology (the sets G1V are viewed as discrete spaces). The set T is closed in 
this product space. We equip T with the induced topology. Then T turns into a 
totally disconnected topological space. Let r = (ti, . . . , tjv) be an arbitrary finite 
path in the graph GT, i.e., t\ G GTi,..., tjv G G1V and t\ -< ••• -< tn- The 
cylinder sets of the form 

C T = {t G T | t\ = ri, . . . ,tN = tn}, 

form a base of topology in T. 

Consider an arbitrary signature A G G1V. The set of finite paths r = (ti -< 
••• -< tn) ending at A has cardinality equal to Dim^ A = x A (e). The cylinder 
sets C T corresponding to these finite paths r are pairwise disjoint, and their union 
coincides with the set of infinite paths t passing through A. 

A central measure is any probability Borel measure on T such that the mass of 
any cylinder set C T depends only of its endpoint A. These definitions are inspired 
by [VK1]. 

There exists a natural bijective correspondence M < — ► {Pn} between central 
measures M and coherent systems {Pn}, defined by the relations 



Dim/v A • M(C T ) = P N (X), 
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where N = 1,2, ... , A G GUV, and r is an arbitrary finite path ending at A. In 
other words, for any N, the pushforward of M under the natural projection 



Y[ GT N DT -> GUV (9.4) 

JV=1 

coincides with Pjv- 

By Proposition 2.1 we get a bijective correspondence between central measures 
and characters of U(oo). This correspondence is an isomorphism of convex sets. 
So, extreme central measures exactly correspond to extreme characters. 

On the other hand, by virtue of Theorem 1.2, we get a bijection M < — > P 
between central measures on T and probability measures on O. In more detail, the 
correspondence M — > P has the form 

M - {P N } ^ X ^P- 

Given a path t, let p^(N,t) and q^(N,t) denote the modified Frobenius coor- 
dinates of the Young diagram (tA r ) ± - We say that t is a regular path if there exist 
limits 



AT^oo N 1 N^oo N N^oo N 

where i = 1, 2, . . . . Then the limit values are the coordinates of a point w G O, and 
we say that uj is the enrf the regular path t or that t ends at u;. 

Let T reg C T be the subset of regular paths. This is a Borel set. Let T Teg — > O 
be the projection assigning to a regular path its end. This is a Borel map. 

Theorem 9.3. Let M be a central measure on T . Then M is supported by the Borel 
set % eg and hence can be viewed as a probability measure on % e g- The pushforward 
of M under the projection T reg — > O coincides with the spectral measure P that 
appears in the correspondence M — > P defined above. 

Proof. See [013, Theorem 10.7]. □ 

Corollary 9.4. Both the measure P and all the measures Pn can be represented 

as pushforwards of the measure M , with respect to the maps 

GTtv, respectively, where the latter map is given by restricting (9.4) to T Teg C T. 

Proof. The claim concerning P is exactly Theorem 9.3. The claim concerning Pn 
follows from the discussion above. □ 

For any compact set A C X, let A/a, at denote the random variable A/a associated 
with the process Vj- N K Here it is convenient to consider as the phase space of 
not the lattice but the ambient continuous space X. Recall that by we 

have denoted the kth correlation measure of V} N ^ . The first step towards Theorem 
9.2 is the following 

Proposition 9.5. Assume that for any compact set Ac X there exist uniform on 
N bounds for the moments of A/a, at , 



n{N A , N ) l ]<c h z = i,2,..., 
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where the symbol E means expectation. 

Then for any k = 1, 2, . . . , the kth correlation measure p k of the process V exists. 
Moreover, for any continuous compactly supported function F on X k = X x • • • x X, 

lim (F,p^} = (F,p k ). 

Proof. We argue as in the proof of [B04, Lemma 6.2]. First, using Corollary 9.4, 
we put all the processes on one and the same probability space, (T reg , M). 
Given a regular path t = (tjv)jv=i,2,...) we assign to it point configurations 

C(t) e Conf(X), C N (t) eConf(X), AT = 1,2,..., 

as follows. 
Let 

u(t) = (a + (t),P + (t);a-(t),p-(ty,S + (t),5-(t)) G O 
be the end of t. Then we set 

C(t) = {at(t) + \} □ {\ - #(*)} U {-aj(t) - \} U {-I + p-(t)}, 

cf. (9.1). Here, as in (9.1), we omit possible O's in a + (t), /3 + (t), a~(t), (3~(t), and 
also possible l's in (3 + (t) or (3~(t). Equivalently, 

C(t) = C(w(t)) = L(w(t)). 

Likewise, for any N = 1, 2, . . . , let p^(N,t) and q^(N,t) denote the modified 
Frobenius coordinates of (^at) 1 * 1 , and let 

afWt) = SM, 6>±wo = ££^i), , ,., 

We set 

C N (t) = {a+(N,t) + \} U {\ - b+(N,t)} U {-aJ(N,t) - f } U {-I + &J(tf,f)}. 

Equivalently, Cjv(t) is the image of under the composite map GTat — > O — > 
Conf(X). 

We view C(t) and Cjv(t) (for any AT = 1, 2, . . . ) as random configurations defined 
on the common probability space (T reg ,M). By Corollary 9.4, these are exactly 
the same as the random configurations corresponding to the point processes V and 
Vs N \ respectively. 

From now on all the random variables will be referred to the probability space 
(T reg , M). Fix a continuous compactly supported function F on X k . It will be con- 
venient to assume that F is nonnegative (this does not mean any loss of generality) . 
Introduce random variables / and /at as follows: 

/(£)= Yl F(x u ...,x k ), f N (t)= F(x u ...,x k ), (9.5) 

x 1 ,...,x k ec(t) xi,...,x fc eCjv(i) 
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summed over ordered /c-tuples of points with pairwise distinct labels. Any such sum 
is actually finite because F is compactly supported and the point configurations 
are locally finite. 

By the definition of the correlation measures, 



(F,p k ) = E[f], (F,pW>=E[/. 



'-k 



N 



and the very existence of pk is guaranteed if E[/] < oo for any F as above. So, we 
have to prove that E[/jv] — > E[/] < oo as A — > oo. By a general theorem (see [Sh, 
ch. II, §6, Theorem 4]), it suffices to check the following two conditions: 
Condition 1. /jv(t) — ¥ f(t) for any t G T Teg . 

Condition 2. The random variables fx are uniformly integrable, that is, 
sup / fN(t)M(dt) — >• 0, as c — > +oo. 

Let us check Condition 1. This condition does not depend on M, it is a simple 
consequence of the regularity property. Indeed, for e > let X e be obtained from 
X be removing the e- neighborhoods of the points ±|, 

X e = X\((-i-e,-i + e)U(|-e,| + e)). 

Choose £ so small that the function F be supported by X*. Fix / so large that 
<x^ < e, (3^ < e for all indices j > I. By the definition of X £ , if a point 

h + <*t(t) 

x=< (9.6) 

-i+/3"(t) 

lies in X e then i < /. 

By the definition of regular paths, for any index i, 

af(N,t)^af(t), bf(N,t)^pf(t), N - oo (9.7) 

Therefore, we have af (A, t) < e and 6^ (A, t) < e for all A large enough. By 
monotonicity, the same inequality holds for the indices / + 1, / + 2, . . . as well. This 
means that if A is large enough and a point 



x = < 



i+a+(A,t) 
l-b+(N,t) 

(9.8) 



i-a~(N,t) 
i + b-(N,t) 

lies in X £ then i < I. 

It follows that in the sums (9.5), only the points (9.6) or (9.8) with indices 
% = 1, ...,/ — 1 may really contribute. By (9.7) and continuity of F we conclude 
that f N (t) - /(*). 
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Let us check Condition 2. Choose a compact set A such that F is supported by 
A k . Denoting by C the supremum norm of F, we have the bound 



f N (t) < C ■ N A , N {t){M A ,N(t) -!)••■ (MaA*) - k + 1) < C ■ (M A At)) k - 

Therefore the random variables f^ are uniformly integrable provided that this is 
true for the random variables {Ma,n) f° r an y fixed k. But the latter fact follows 
from the assumption of the proposition and Chebyshev's inequality. □ 

To apply Proposition 9.5 we must check the required uniform bound for the mo- 
ments. By the assumption of Theorem 9.2, each point process V_ N is a determinantal 
process on X n such that its kernel, restricted to xj^ x X_[^ and to X ^ t x X™ ut , is 
Hermitian symmetric. Here we set X^ = X^ n Jwt = X^-* ^ ^out- For a 
compact set A C X we denote by K_ A the restriction of the kernel to A n X} N ^ . If 

A is entirely contained in £ in or X ou t then is a finite-dimensional nonnegative 
operator. 

Proposition 9.6. Assume that for any compact set A, which is entirely contained 
in Xi n or X ou t, we have a bound of the form tr K_ A < const, where the constant 
does not depend on N . Then the assumption of Proposition 9.5 is satisfied. 

Proof. Let A C X be an arbitrary compact set. Then A = A- in U A out , where 
A- in C Xi n and A out C X out are compact sets. We have Na,n = J^A OVLt ,N +J^A in ,N- 
If we know uniform bounds for the moments of Af Aout ,N and AfA in ,N then, using 
the Cauchy-Schwarz-Bunyakovskii inequality, we get such bounds for Na,n, too. 
Consequently, we can assume that A is entirely contained in X ou t or Xi n , so that 
K_ A is nonnegative. 

Instead of ordinary moments we can deal with factorial moments. Given / = 
1,2,..., the Z-th factorial moment of Ma,n is equal to 

p\ N \A l )= [ det\Kf\x il x j )] 1 < iJ < l dx 1 ...dx l = l\tT(A l K A N) ). 

~ L J A' 

Since K_ A is nonnegative, we have 

tr(A^) < tr(®'£if°) = (tr(£<f°))'. 

This concludes the proof, because we have a uniform bound for the traces by as- 
sumption. □ 

Proof of Theorem 9.2. We shall approximate the process V by the processes v}- N \ 
Recall that the process p( N ) is a scaled version of the process J>( N \ and their 
correlation kernels, I(} N ^ and K^ N \ are related as follows 

jr (JV) (x, y) = N ■ KW (Nx, Ny), x, y G X (A ° = ±X. 

Let us check that the assumption of Proposition 9.6 is satisfied. Without loss of 
generality we may assume that A is a closed interval [a, b], contained either in Xi n 
or Xout- We have 

tr ^° = ^ E K W (x,x), 
xeAnX" 
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where the factor comes from the reference measure p^ on the lattice %( N ) 
(recall that p^ assigns weight to each node). By the relation between the two 
kernels, this can be rewritten as follows 

trK^ = ^ J2 N.K^ N \Nx,Nx). 

xeAnx n 

By the assumption of Theorem 9.2, the right-hand side tends, as N — > oo, to the 
integral f a K(x,x)dx, hence the traces above are uniformly bounded, as required. 

Proposition 9.6 makes it possible to apply Proposition 9.5. It shows that the 
process V possesses correlations measures pk, which are weak limits of the measures 

On the other hand, we know that for any k, the correlation measure p^ is ex- 
pressed, by a determinantal formula, through the correlation kernel K}- N \ Further, 
by the assumption of Theorem 9.2, this kernel tends to the kernel K as N — > oo. 
This implies that the limit measure pf~ is expressed, by the same determinantal 
formula, through the limit kernel K. □ 

Now we return to the discussion of the correspondence P \— > V. Recall that it is 
based on the map i : Q — > Conf(X), sending a point u = (3 + ; a~ , (3~;5 + ,5~) to 
a configuration C(u). This is a continuous analog of the map GTn — > Conf(X^ Ar ^) 
sending a signature A to a point configuration X(X), see (4.1). The correspondence 
A i— > X(X) is reversible while the map u> h- > C(u) is not. This is caused by three 
factors listed below. 

(i) The coordinates (3f and (3~ become indistinguishable: given a point x £ 
C(u) fl Xi n , there is no way to decide whether it comes from a coordinate \ — x of 
(3 + or from a coordinate x + | of (3~ . Note that in the discrete case such a problem 
does not arise. Indeed, if d + and d~ stand for the numbers of points x G X(X) that 
are on the right and on the left of X; n , respectively, then X(X) has exactly d + + d~ 
points in X; n of which the leftmost d~ points come from A - while the remaining d + 
points come from A + . But in the continuous case, such an argument fails, because 
the total number of points is, generally speaking, infinite. 

(ii) The map t ignores the coordinates 5"^. 

(iii) The map i ignores possible l's in (3^. 

Let us discuss the significance of these factors in succession. 
Factor (i). Remark that exactly the same effect of mixture of the plus and minus 
/^-coordinates arises when an extreme character 

x («) i s 

restricted from the group 

U(oo) to the subgroup SU (oo), see [013, Remark 1.7]. Hence, if one agrees to view 
characters that coincide on SU(oo) as equivalent ones, then the factor in question 
becomes not too important. 

Factor (ii). We conjecture that in our concrete situation (i.e., for the characters 
Xz,z',w,w') the spectral measure P is concentrated on the subset of cu's with 7 ± = 
(see the definition of ^ in §1). If this is true then 5^ is almost surely equal 
to XX + @t)- The conjecture is supported by the fact that the vanishing of 
the gamma parameters was proved in similar situations, see [P.I, Theorem 6.1] 
and [B04, Theorem 7.3]. The method of [B04] makes it possible to reduce the 
conjecture to the following claim concerning the first correlation measures of the 
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processes V_ 



lim / \x-\\ p ( 1 7V) (dx) = 0, lim / \x + \\ p}^ (dx) = 
uniformly on N. 

Factor (iii). Again, we remark that possible l's in (3^ play no role when char- 
acters are restricted to the subgroup SU (oo). On the other hand, we can prove 
that our processes, one can speculate that in concrete the l's do not appear almost 
surely. 

The above arguments (though not rigorous) present a justification of the passage 

We conclude the section with one more general result which will be used in §10. 
Observe that the set of characters of U(oo) is stable under the operation of 
pointwise multiplication by det( • ) fc , where fcsZ. 

Proposition 9.7. Let x be a character ofU(oo), P be its spectral measure on tt, 
and V be the corresponding point process onX = l\{±|}. Then V does not change 
under x l—> X det( • ) k , k e Z. 

Proof. It suffices to prove this for k = 1. Assume first that x is extreme, so that 
X = X -i where to = (ct + , a~ 5 + , S~) G O, and P reduces to the Dirac mass 
at {uj}. From the explicit expression for x^\ see 

(1.2), it follows that x {uj) det( •) 
is an extreme character, too. Moreover, the corresponding element w60 looks as 
follows: the parameters a^ 1 and do not change, while 

0+ _> (i _ /?-,/?+, /?+,...), p~ » (/? 2 -, fe, . . . ). 

On the other hand, from the definition of the projection u \— > C(uj), see (9.1), it 
follows that the change u i— > u does not affect the configuration C(u). This proves 
the needed claim for extreme x- 

Now let x De arbitrary. By the very definition of spectral measures (see Theorem 
1.2), the spectral measure of the character x det( • ) coincides with the pushforward 
of the spectral measure P under the map u i— > u of O. We have just seen that this 
map does not affect the projection u> i— > C(u). Since V is the image of P under this 
projection, we conclude that V remains unchanged. □ 



10. The correlation kernel of the process V 



Our goal in this section is to compute the correlation functions of the process V 
associated to the spectral measure for zw-measures, see the beginning of §9 for the 
definitions. Theorem 10.1 below is the main result of the paper. 

In the formulas below we use the Gauss hyper geometric function. Recall that 
this is a function in one complex variable (say, u) defined inside the unit circle by 
the series 

a, b 



u 



= E 



fc>0 



(a)k(b) k k 
k\{c) k • 



Here a, 6, c are complex parameters, c ^ {0, —1, . . . }. 
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This function can be analytically continued to the domain u G C \ [l,+oo), 
see, e.g., [Er, Vol. 1, Chapter 2]. We will need the fact that for any fixed u in 
C \ [1, +oo), the expression 



1 



r(c) 



a, b 
c 



u 



defines an entire function in (a, 6, c) G C 3 . This follows, e.g., from [Er, 2.1.3(15)]. 

Theorem 10.1. Let (z, z', w, w') G £> a dm and V be the corresponding point process 
on X = R \ {±^} defined in §<?. 

For anyn = 1,2,... and x±, . . . , x n G X, the nth correlation function p n (x\, . . . , x n ) 
of the process V has the determinantal form 

p n {x u . . . , x n ) = det[K(xi, Xj)]ij =1 . 

The kernel K(x, y) with respect to the splitting X = X out U Xi n has the following 
form 



K out , out (x,y) = y/i/) ut(x)i/>out(y) 



Rout(x)S out (y) 


- S out (x)R out (y) 


X 


- y 


R out (x)R in (y) - 


- S out (x)S in (y) 


x - 


i 

-y 


R in (x)R out (y) - 


- S in (x)S out (y) 


x - 


1 

- y 


Rin(x)S in (y) - 


S in (x)R in (y) 



x-y 



where 



C{z,z')-{x-\Y z - z {x + \) — 
C(w,w')-(-x-±)- w - w ' (-x+J) 

^in(x) = {\~ X) Z+Z {\ + x) W+W 



X > 2 , 



, X < 2 > 



2 < X < 2 , 



sin ttz sin 7rz . sin 7r«; sin 7rur 

(7(z,z) = 5 , C(w,w) = - 



7T 



7T Z 



and 



flout (a) = 

Sout(x) = r 



X — 2 



^1 



z + u/, z' + u/ 
2 + z' + w + w' 



2 X 



z + w + 1, z + w' + 1, z' + w + 1, z' + w' + 1 
z + z' + «; + «/ + 1, z + z' + w + w' + 2 



1 



x 2 \ x 2 



X + 2 



z + w' + 1, z' + «;' + 1 
z + z' + w + w' + 2 



— x 
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Rin(x) = ~ 



X 



smnz 

TV 



+ X 



z' — z, z + w + 1, z + w' + 1 
z + w + z' + w' + l 



1 



— X 



z + w' + 1, —z' — w 
z - z' + 1 



— X 



X 



SIIITTZ 



TV 

1 

2 +X 



z - z', z' + w + 1, 2/ + w' + 1 
z + tu + z' + iu' + l 



5"* 



2/ + «/ + 1, 



to 



2 + 1 



1 

2" X 



Sin (a) = 



X 



Sin7T2 



7T 



- z, z + z' + w + w' 
z' + to, z' + w' 



1 

2~ X 



z + w', —z' — w + 1 
z-z' + l 



2~ X 



simrz 

TV 



z — z', z + z' + w + w' 
z + w, z + w' 



X 



1 



— X 



z' + w', — z — w + 1 
z' - z + 1 



1 



— X 



The indeterminacy on the diagonal x = y is resolved by the L 'Hospital rule: 

,out 

K in , in (x,x) = iJi n (x) (R[ n (x)S in (x) - S[ n (x)R in (x)) . 

Singularities. The formulas for the function -Rout, S out , Rim Sin above have no sin- 
gularities for 

(z, z', w, w') EV \ ({£ = 0} U {z - z' g Z}) . 

Moreover, we will prove that the value of the kernel K{x, y) can be extended to a 
continuous function on £> a dm for any fixed x,y G X. (Recall that the process V is 
defined for (z, z', to, w') G "P a dm-) 

Vanishing of the kernel. Note that if (z, z') G i^degen (see §3 for the definition of 
Z&egen) then the function ip out vanishes on (|,+oo), because C(z,z') = 0. This 
implies that K{x,y) = whenever x or y is greater than |. It follows that the 
configurations of the process V do not have points in (|, +00). 

Likewise, if (to, to') G ^degen then the configurations of the process do not inter- 
sect (—00, — |). 

Proof. First of all observe that it suffices to prove the theorem when (z, z', w, w') G 
^adm- Indeed, if (z, z', w, w') G "D a d m \^ dm then (z, z', w, w') can be moved to 
^adm by an appropriate shift of the parameters, which is equivalent to multiplying 
the initial character x by det( • ) fc with a certain fceZ, see Remarks 6.4 and 3.7. 
Next, according to Proposition 9.7, multiplication by det(-) fc does not affect the 
point process V. 

To carry out the desired reduction we have to check that the formulas for the 
functions p n given in Theorem 10.1 are also invariant under any shift of the param- 
eters of the form 



(z, z' , to, w') 1— > (z + k, z + k,w — k,w' — /c), k G Z. 
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Note that the kernel K(x, y) is not invariant under such a shift. To see what 
happens with the kernel we observe that 

1 . 2k / . , 1 x ~ 2k 



flout fa) -> ^ ~ _ | J Rout(x), S out (x) -> ^ - - f j Soutfa), 



2 / \~ 2 

fc , i x k 



(1 \ / 1 \ 

£ + 2 / V ^ + 2 / 



It follows that 
where 



Kfa,y)^0fa)Kfa,y)fa(y))-\ 
0fa) = 



1, x G X ou t, 

( 1) 5 £ G Xj n . 

But such a transformation of the kernel does not affect the determinantal formula 
for the correlation functions. 

From now on we will assume that (z, z' ,w,w') G T>' &din , as m Theorem 8.7. At 
this moment we impose additional restrictions E ^0 and z — z' ^ Z. We will need 
these restrictions in Propositions 10.3-10.4 below. After that they will be removed. 

For any i6l = R\{±|}, let xn denote the point of the lattice = Z+ 
which is closest to Nx (if there are two such points then we choose any of them). 
By Theorems 8.7 and 9.2, it suffices to prove that 

lim N-K( N \x N ,y N ) = K(x,y), (10.1) 

N^oo 

uniformly on compact sets of X x X. Here K( N > is the kernel of Theorem 8.7. 
To do this we will establish the uniform convergence of all six functions Vw > 

\ Rout i Sout i Mn i of Theorem 8.7 to the respective functions of Theorem 
10.1. In order to overcome the difficulty arising from vanishing of the denominators 

at x = y we will establish the convergence of -R^ut > ^out > -^in > &in m a complex 
region containing X. 

The needed convergence (10.1) follows from Propositions 10.2-10.4 below. 

Proposition 10.2. There exist limits 

lim iV^Sfatf) = Vwtfa), Jim N'^ip^' (x N ) = ^(x), 

as N — > oo, where the functions ip^ an< ^ were defined in (6.2), (6.3). The 

convergence is uniform on the compact subsets of X out = K \ [— |, \] and X; n = 
respectively. 

Proposition 10.3. Let I be a compact subset o/X ou t- Set 

/ £ = {CgC|KCg/, I SCI <e}- 
Then for e > small enough, for any ( G I s we have 



and the convergence is uniform on I t 



Jim CW) = ^out(C), Jim N^S^m) = S out ((), (10.2) 
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Proposition 10.4. Let J be a compact subset of X iri . Set 

J £ = {(eC\K(eJ,\Z(\<e}. 

Then for e > small enough, for any ( G J e we have 

lim R^\n() = i? in (C), lim N*S^(N() = S in (0, (10.3) 
and the convergence is uniform on J e . 

Proof of Proposition 10.2. Follows from the following uniform estimates. 
For x G Xj n we have 

r(- XN + u + ^±i) ( ^ i 



T(x N + v + ^) 

T( XN + m 



N u (-x + ^J (1 + 0(N~ 1 )), u = zorz, 
= N V (x+^J (! + 0(N- 1 )), v = wotw', 



as N — > oo. 

For x G X ou t and x > | we use the formulas 



^iv+ 2 — [x N -u — ), u = zotz', 



sin(-7ra) / N — 1 



and the asymptotic relations 



T(x N -u-^) Ar _ / 1 



r( X7V - ^) 

r(x N + z±±) „_., / . i 



2/ =AT- u ^r-l^ (1 + 0(A^ -1 )), u = zor 2 ', 



r(x w + H^) 



^""(^+7^ (! + 0(N~ 1 )), v = worw', 



as iV — > oo. 

For x G Xout and x < — | we use the formulas 



1 sin ^(a^ + v + ^)) / N — 1 

X N — V 



F(x N + v+^) 

= (-1)^+ 2 5: — Lri-x N -v — J, v = w or w, 



and the asymptotic relations 



N-»(-x+l) (1 + OiN- 1 )), u = zovz>, 
2 ^=N- v (-x-^J (l + OiN' 1 )), v = wovw', 



T(-x N -v-^ ) Ar _„ ( P 

r(-a* - ^) 
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as N — > oo. □ 

Proof of Proposition 10.3. We will employ the formulas (8.1) and (8.2). We have 



j , N+l 



N(-^,N( + w' + , 

NC+ N±l^ N( + w ,_N-L 



3-^2 



—N, z + w', z' + w' 
E, NC + w'- 



Assume 3?C < — §• Handling the gamma factors is easy: 



iVC + ^, NC + w'- 2 



,/ N-l 



r(-N( 



N-l ■ 



r(-NC-w-*=±) r(-ivc + ^i) 



(1 + OiN- 1 )) 



c+j 
c-l 



(1 + OiN- 1 )), 



as N — > oo, and the estimate is uniform on a small complex neighborhood of any 
compact subset 7 C (— oo, — |). 

To complete the proof of the first limit relation (10.1) we need to have the 
equality 



lim 3 F 2 

TV— >oo 



—N, z + w', z' + w' 
z + z' + w + w' , N( + w' - ^f 1 



1 



= 2F1 



z + w', z' + w' 
z + z' + w + w' 



1 



with the uniform convergence. This limit transition is justified by the following 
lemma. 

Lemma 10.5. Let A, B, C be complex numbers, C^O, —1, —2, . . . , {-D/v}/v=i be 
a sequence of complex numbers, Dn 7^ 0, — 1, . . . for all N. Assume that 



lim 

N^oo D 



-N 



Then 



lim 3 F 2 



N 



-N, A, B 
C, D N 



qe (0,1). 



2-^1 



A, B 
C 



The convergence is uniform on any set of sequences {-Dat}^ =1 such that {75^- — 
lim-^-} uniformly converges to as N — > 00 and lim is uniformly bounded 
from —1 on this set. 

Proof. We have 



3-^2 



-N, A, B 



C,D 



N 



1 



N 



^ Q k\(C) k (D N ) k 



Let us show that these sums converge uniformly in N > No for some Nq > 0. 
Thanks to our hypothesis, for large enough N we can pick q e (q, 1) and a positive 
number ei/v such that for all our sequences {D N }, -$W N > d N > Nq^ 1 . Then 
(note that k < N) 



[D N ) k \ = \(-D N )(-D N - 1) . . . (-D N - k + 1)| > d N (d N - 1) • • • (d N - k + 1). 
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Hence, 



(-AO/ 



(D N ) f 



< 



N(N - 1) • • • (N - k + 1) 



< 



N 
d N 



<Qo- 



d N (d N - 1) • • • (d N - k + 1) 
Thus, the sums above for large enough N are majorated by the convergent series 

(A) k (B) k 



E 



fc=0 



(to* 



k\(C) k 

and, therefore, converge uniformly. This means that to compute the limit as N — > 
oo we can pass to the limit N — > oo in every term of the sum. Since for any fixed k 

(-N)k u 



lim 



{D N ) k 



this yields 



lim 3 F 2 



N^oo 



-N, A, B 
C, D N 



^ (A) k (B) k gfc 



fc>0 



k\(C)> 



A, B 
C 



The fact that we majorated the series by the same convergent series for all our 
sequences {-Dtv}, and the uniform convergence of the terms of the series guarantee 
the needed uniform convergence on the set of sequences. □ 

To prove the first limit relation for i" C (\, +oo) we just note that by uniqueness 

of monic orthogonal polynomials with a fixed weight, R^(x) is invariant with 
respect to the substitution 



x 



-x, (z,z')< — >(w',w), 



A, B 
C 



c 



= (i-C) 



- A TP 



~A,C-B 


C 1 


C 


C-iJ 



The proof of the second relation (10.1) is very similar. Note that both S^2(x) 
and S ou t(x) are skew-symmetric with respect to the substitution above. 
The proof of Proposition 10.3 is complete. □ 

Proof of Proposition 10. 4- The argument is quite similar to the proof of Proposition 
10.3 above. Let us evaluate the asymptotics of the right-hand side of (8.4). Let us 
look at the first term. Clearly, the argument for the second term will be just the 
same. Gamma-factors give (here £ J C (- \->\)) 

N( + ^±i, -N£ + ^±i, N + 1 + E 
-N( + z'+ ^±1, NC + w + ^,N + l + z + w' 



(l + OtiV" 1 )), 



as N — > oo, uniformly on a neighborhood of J. To complete the proof of the first 
relation (10.2) we need to show that 

,. „ \z + w' + l, -z'-w, -N( + z + ^ 

N^oo* Z-z' + l, N+l + Z + w' 



1 



= 2F1 



z + w' + l, -z' 
z-z' + l 



w 



1 



-c 



uniformly in (. This is achieved by the following lemma. 
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Lemma 10.6. Let A,B,C,5 be complex numbers, C ^ 0,-1,-2,..., {-Dtv}^ =1 
be a sequence of complex numbers. Assume that 

lim ^ = qe (0,1). 



Then 



lim 3 F 2 

TV^oo 



'A, B, D N 
N + 5,C 



— 2F1 



A, B 
C 



The convergence is uniform on any set of sequences {-Djv}jv=i such that {^f- — 
lim-^-} uniformly converges to as N — > 00 and lim 5^ is uniformly bounded 
from 1 on this set. 



Proof. We have 



3-^2 



A, B, D N 
N + 5,C 



= E 

fc=0 



(A) k (B) k (D 



Nik 



k\(N + 8) k (C) k 



Let us show that these sums converge uniformly in N > No for some No > 0. 

Let dw be the smallest integer greater than sup|-Djv|, where the supremum is 
taken over all our sequences {D^}. Let / be the largest integer less than JW. Then 
for large enough N our series is majorated by the series 



E 

fc=0 



(A) k (B) k (d N ) k 



k\(N + l) k (C) k 



Using the hypothesis we may assume that for large enough N, dN < qo(N + 1) for 
some qo € (0, 1). In particular, <i/v < N + I. If k < N + I — dN then we have 



{d 



Nik 



d N (d N + l)---(d N + k-l) 



(N + l) k (N + l)(N + l + l)---(N + l + k-l) 

k 

I (I \ 4- I V -:-/■■;/ v ! \ 



< 



d N + (N + I - d N ) 



N + l 



N + l + (N + l-d N )J \2N-d N + 2l 
for large enough N. If k > N + I — d]^ then we get 

(d N ) k d N (d N + 1) • • • (d N + k - 1 



d N + k - 1 
N + l + k-1 

<(2-q )- k 



(N + l) k (N + l)(N + 1 + 1) ■■■(N + l + k-1) 



d N (d N + l)---(N + 1-1) 



(d N + k)(d N + k + 1) • • • (N + I + k - 1) 

k N -(N+l-l-d N ) 



< 



N + l 



N+l-l-d N 



< 1 + 



N + l 



< 1 + 



N + l + k / 
k \ -(l-3o ) v 



iV + Z 



for large enough N. The last expression is a decreasing function in N (assuming 
N + l>0). Hence, for N > N we get 



Nik 



(N + l) k 



< 1 + 



fc 



iVo + / 
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-(l-«o)(iVo+i) 



Thus, we have proved that (djv)fc/(-^ + l)k does not exceed the maximum of 
the kth member of a geometric progression with ratio (2 — Qo) 1 < 1 an d the 
inverse of the value at the point k of a polynomial of arbitrarily large (equal to 
[(1 — qo)(N + /)]) degree. Since (A) k (B) k /(k\(C) k ) has polynomial behavior in k 
for large k, this means that we majorated the series 



3-^2 



A, B, D N 
N + 5,C 



{A) k {B) k {D N ) k 
^ k\(N + 5) k (C) k 



E 



fc=0 



by a convergent series with terms not depending on N. Therefore, to compute the 
limit of this series as iV — > oo, we can take the limit term- wise. Since for any fixed 
k > 

{D N )k * 
Q i 



lim 



iv^oo (N + 6) k 



we get 



lim \ 



{A) k {B) k {D N ) k 



= E 



v^oo^fc!(JV + 5) fc (C) fc ^ fc!(C) fc 



(i4) fc (S)* fc 

9 — 2-^1 



C 



As in the proof of Lemma 10.5, the fact that we majorated the series by the same 
convergent series for all our sequences {Dn}, and the uniform convergence of the 
terms of the series guarantee the needed uniform convergence on the set of se- 
quences. □ 

The proof of Proposition 10.4 is complete. □ 

To conclude the proof of Theorem 10.1 we need to get rid of the extra restrictions 
E 7^ and z — z' ^ Z imposed in the beginning of the proof. 

Define a function : X — > C, which is similar to the function ty( N ^ introduced 
in (8.10), by 

{Vwt(aO, x G X out , 
_L_ xeX . 

Note that for any iGl, ty(x) is an entire function in (z, z' ', w,w'). 

Lemma 10.7. Let (z, z' ,w,w') G L)' adrn . The kernel K(x,y) can be written in the 
form 

K(x,y) = y/V(x)*(y)K(x,y), 



where K(x,y) admits a holomorphic continuation in the parameters to the domain 
V' d D^ dm . Moreover, for any (z, z', w, w') G V , 

K(x,y)= lim N^K^ix^VN) 



uniformly on compact subsets of X x X. 

o 

Recall that the kernel if (Ar) was defined in Lemma 8.8 and V was defined just 
before this lemma. 
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Proof of the lemma. It will be convenient to use a more detailed notation for the 

o 

kernels in question. So, we will use the notation K^ N \x, y \ z, z' , w, w') instead of 

o o 

K( N ) (x, y). Next, we define the kernel K(x, y \ z, z' , w, w')\ in the block form, 

° f , / ,v Ront{x)S ont (y) - S out {x)R out {y) 

K outjOUt (x,y z,z ,w,w ) = , 

x — y 

K out ,in(x, y z,z ,w,w) = , 

x — y 

R i n (x)R out (y) - S in (x)S OVLt (y) 
x-y 

Rin{x)S in (y) - S in (x)R in (y) 



K in!OUt {x, y | z, z', w, w') = 

o 

K- in - in {x,y | z, z',w,w') = V(x)V(y) 



x-y 

These expressions are well defined if (z, z' ', w, w') is in the subdomain 

V ' = {(z, z ', w, w') G V' | E ^ 0, z - z <£ Z} 
By virtue of Propositions 10.2, 10.3, and 10.4, 

lim N^K^ (x N , y N \ z, z', w, w') = K(x, y | z, z' , w, w') (10.4) 

for any fixed (z, z', w, w') G Vq , uniformly on compact subsets of X x X. Moreover, 
one can verify that the estimates of Propositions 10.3 and 10.4 are uniform in 
(z, z' ,w,w') varying on any compact subset of the domain Vq. Thus, the limit 
relation (10.4) holds uniformly on compact subsets of X x X x Vq. 

o 

On the other hand, we know that the kernel K^ N > is holomorphic in (z, z', w, w') 
on the larger domain V' D Vq. It follows that the additional restrictions E ^ 
and z — z 1 ^ Z can be removed. Specifically, the right-hand side of (10.4) can 
be extended to the domain V' and the limit relation (10.4) holds on X x X x V' . 
Indeed, we can avoid the hyperplanes E = or z — z' = k, where k G Z, by making 
use of Cauchy's integral over a small circle in the z-plane. 

This completes the proof of Lemma 10.7. □ 

Now we can complete the proof of the relation (10.1). Proposition 10.2 implies 
that 

lim N j: q (N) (x N ) = y(x) (10.5) 

for any (z, z', w, w') G V' Q , uniformly on compact subsets of X. Indeed, as is seen 
from the proof of Proposition 10.2, it does not use the additional restrictions and 
holds for any (z, z', w, w') G Vq. To pass from the functions Vwt ) V'in > an< ^ ''/'out, 
ipin to the functions ty( N > and we use the assumption (z, z' \w,w') G V' which 
makes it possible to invert the 'inner' functions for all values of parameters. (Note 
that if {z, z') G Z& egen then for x > \ and large enough N both ^ n \xn) and ^(x) 
vanish. Similarly, if (w,w') G ^degen then the vanishing happens for x < — |.) 
Since 

K(x,y) = k(x,y)y/V{x)V(y), 



KW (x, y) = K (A ° {x, y) ^( N ) {x)^( N ) (y), 
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(10.1) follows from (10.4) and (10.5). This completes the proof of Theorem 10.1. □ 

We conclude this section by a list of properties (without proofs) of the correlation 
kernel K and functions -R ou t, Sout, -Rin, S\ n . The proofs can be found in [BD]. 

All the results below should be compared with similar results for and R^t > 
S^t > Rin\ &in > which were proved in the previous sections. 

Symmetries. All four functions -R ou t, Sout, -Rin, 'S'in are invariant with respect to 
the transpositions z <-> z' and w <-> w'. 

Further, let us denote by S the following familiar change of the parameters and 
the variable: (z, z', w, w', x) < — ► (w, w', z, z', —x). Then 

S(lp out ) = Ipout, S(lp in ) = lp in , 
<S(Rout) = -Rout, <S(S ou t) = — S ou t, S(Rin) = -Rin, S(Si n ) = —Si n . 

The functions R ou t, S out , _Ri n , Si n and the kernel K for admissible values of 
parameters take real values on X. Moreover, the kernel K(x, y) is J-symmetric, see 
§5(f). That is, 

^out,out(^,y) = ^out,out(y,aO, -K"in,in(^,y) = K iQ , in (y,x), 

Ki niOUt (x,y) = -K mttin (y,x). 

Branching of analytic continuations. The formulas for i? ut, S ou t, -Rin, Sin 
above provide analytic continuations of these functions. We can view -R ut and 
.S^t as functions which are analytic and single- valued on C \ £j n , and R[ n and 
S- m as functions analytic and single- valued on C \ X out . (Recall that the Gauss 
hypergeometric function can be viewed as an analytic and single valued function 
on C\ [l,+oo).) 

For a function F(Q defined on C\R we will denote by F + and F~ its boundary 
values: 

F + (x) = F(x + i0), F~(x) = F(x - iO). 



Then we have 



1 S^-, + „ 1 -R„„j- R, 



+ 



<vr 2 out ^out t-> ''out ''out rt 

Oil Xi n -fi-in , , „ '-'in , 

^i n 2m ip- m 2m 

i q— c+ i p— p+ 

x J in °in p in - rt in c 

on x ou t ~ — : — -n, ou t , ~, ~ — : — <->out • 

V'out 2ttz f out 2tu 

This can also be restated as follows. Let us form a matrix 



m = 



-Rout Sin 
Sout -Rin 



Then the matrix m has the jump relation m + = m-V on X, where the jump matrix 
equals 

1 27TZ^out(^)\ 

q ^ J 1 X ^ -tout) 



1 <»'• 

27TZ ^ in (x) 1 ' 



Furthermore, if E > then m(£) ~ I as £ — > oo. 
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Differential equations. We use Riemann's notation 







oo 


i 

2 




w 


1 




a: 


\ w' 


— E 


z' 




1 

2 

-w' 


oo 
1 


1 

2 


X 


—w 


E 


— Z 





to denote the two-dimensional space of solutions to the second order Fuchs' equa- 
tion with singular points ti,t2, £3 and exponents a, a'; 6, b'; c, c', see, e.g., [Er, Vol. 
1,2.6]. 
We have 

Rout(x) e P \ w z x I , S ou t(aO G 
w' 

Rin(x) e P ( -«/ 

—to 

The resolvent kernel. There exists a limit 

L(x,y)= lim AT • (xjv, dn) , x.yEdC. 

In the block form corresponding to the splitting X = X ou t U Xi n , the kernel L(x,y) 
looks as follows: 

[ .A" 
|_-„4* 0_ 

where ^4 is a kernel on X out x Xj n of the form 



x - J/ 



This kernel defines a bounded operator in L 2 (X, fix) if and only if |z + z'\ < 1 
and \w + w'\ < 1. If, in addition, we know that E > then we can prove that 
L = Kj (1 — K) or K = L/(l + L) as bounded operators in L 2 (X, dx). 

11. Integral parameters z and w 

If one of the parameters z, z' and one of the parameters w, w' are integral then 
the measure Pn defined in §3 is concentrated on a finite set of signatures, and there 
is a somewhat simpler way to compute the correlation kernel of V. 

Let us assume that z = k and w = /, where k, I e Z, k + l > 1. Then (z, 2/, «/) 
form an admissible quadruple of parameters (see Definition 3.4) if z' and w' is real 
and 2/ — k > —1, w/ — Z > —1. We excluded the case k + l = from our consideration 
because in this case the measure Pn is concentrated on one signature. 

It is easily seen from the definition of Pn that the measure Pn is now concen- 
trated on the signatures A G GT N such that 

k > Ai > ••• > Ajv > -I. 
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Note that it may happen that this set does not include the zero signature because 
k and — / can be of the same sign. 

From now on in this section we will assume that A satisfies the inequalities above. 
Denote 



x fc,z — S 2 '»•••) 2 
Then 

£(A) = {A 1 -l + ^±i,...,A JV -iV + ^±i}c4?. 
Let us associate to A a point configuration Y(X) in %( N ) as follows 

y(A)=X^ ) \£(A). 

Note that Y(X) defines A uniquely. Since |£(A)| = N, we have |Y(A)| = k + l. Let 

Y W = {yi,---,yk+i}- 

The configuration Y(X) coincides with the configuration X(X) = £(A) A from (4.1) 
on the set fl ■ 

Proposition 11.1. Let z = k, w = I be integers, k + l > 1, and z' > k — 1, 

w' > I — 1 be real numbers. Then 

.... «r(- s ,+/ + ffi]r(s i + »-'+fli) _ 

Pn{x) = const 5 wWt#) 1S n t+ <* - *> ■ 

Remark 11.2. Note that for any integer n the shift 

k k + n, I i— > / — n, z' i— > z' + n, it?' i— > u/ — n, y i— > y + n 

leaves the measure Pjv invariant, cf. Remark 3.7. This means that essentially the 
measure depends on three, not four, parameters. If we now set / = then A can be 
viewed as a Young diagram. Then one can show that Y(X) = {^-^- — Xj + j}j =1 , 
where A' is the transposed diagram. 

Proof of Proposition 11.1. Set Xi = Aj — % + — Then by Proposition 6.1 

N 

P N (X) = const Y[ f(xt) | ] (xi - Xjf. 

i=l l<i<j<N 

Then, since {yi}i=i = %ki \ { x i}iLii similarly to Proposition 5.7 we get 

N 

P N (X) = const Y[ KVi) II (Vi ~ Vj) 2 y 
i=l i<i<j<k+l 

where 



N — 1 . N — 1 



f(y) n (?/-*) 2 
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Substituting 



xex y k y\ y 
and f(x) from (6.1) we see that 



/ , N+l ■ 



T( - V + Z ' + H±± nv + W > + , ; 
w rf-j + fc + ^jrfa + i + iai) ' 

Denote by V^i the point process consisting of the measure Pat (A) on point 
configurations Y(X). 

Below we will be using Hahn polynomials. These are classical orthogonal poly- 
nomials on a finite set, and we will follow the notation of [NSU]. 

Proposition 11.3. For any n = 1,2, ... , the nth correlation function of the pro- 
cess vffl has the form 

P i N Hyi,...,yn) = det[Ki N l \y i ,y j )]l j ^. 

-KfcP is the normalized Christoffel-Darboux kernel for shifted Hahn polynomials 
defined as follows: 

where m = k + 1, h(x) is as above, 

^ m {x) = h£- k ' w '-» (x + l+ ^± m + N^j, 
?m-i(x) = h%- 1 k > w '- l) (x + l+ m + N^j 

are Hahn polynomials, 

H m -\ 



and the numbers A m -i, A m are the leading coefficients of h^_ k,w ^ (x,m + N) 
and hm k,w ^ (x, m + N) . 

Proof. Note that if we shift our phase space by / + ^f^- then the weight 

function turns into the function 

j ( , N-l\ F(-y + I + z' + N)T(y - I + w' + 1) 
h[y—l— 

on the space 



2 ) T(-y + k + I + N)T(y + 1) 



4? + 1 + = {0,l,...,k + l + N-l}. 
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But this is exactly the weight function for the Hahn polynomials 
h^'- k ^'- l \y,k + l + N), n = 0,1,2,..., 
see [NSU, 2.4]. Then the claim follows from Proposition 5.1. □ 

Explicit formulas for the Hahn polynomials and their data can be found in [NSU] . 

We know that the processes 7>W and restricted to the set x|f } n 

coincide by construction. Same is true for the correlation kernels, but it is not 
obvious (recall that the correlation kernel of a determinantal point process is not 
defined uniquely, see §5(b)). 

Proposition 11.4. For any x, y G n %*k? > 
Proof. Follows from the relations (here x G Xj^P) 



^ m _ 1 (x)v / ^I= (-l) x - k - E ^p N (x)y/J(^), 
A m -i = tlN, A m = flN-l, H-m-1 = h>N- 

(The polynomials pjv-i(a;), Pn(x) were introduced in (7.1).) 

These relations can be proved either by a direct verification using explicit for- 
mulas (which is rather tedious), or they can be deduced from the following general 
fact. 

Lemma 11.5 [B5]. Let X = {xq,xi, . . . ,xm} be a finite set of distinct points on 
the real line, u(x) and v(x) be two positive functions on X such that 

u{x k )v{x k ) = = — k=0,l,...,M, 

and Pq, Pi, . . . , Pm and Qo, Qi, . . . , Qm be the systems of orthogonal polynomials 
on X with respect to the weights u(x) and v(x), respectively, 

degPi = degQi = i, \\Pi\\ 2 = p^ \\Qi\\ 2 = qi, 
Pi = aiX % + lower terms, Qi = biX % + lower terms. 

Assume that the polynomials are normalized so thatpi = qu-i for alii = 0,1, ... , M. 
Then 

Pi(x)yju(x) = e{x)Q M -i{x)\/v(x), xeX, 
dib M -i = Pi = qM-i, i = 0, l,...,M, 



where 



e{x k ) = sgn JJ(x fc - Xi), k = 0,l,...,M. 



Taking 



M = N + m-l, X = X ( k N l \ u(x)=f(x), v(x) = h(x) 
we get the needed formulas. The proof of Proposition 11.4 is complete. □ 
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Theorem 11.6. Assume z = k and w = I are integers, k + I > 1, z' and w' are 
real numbers such that z' — k > — 1, w' — I > — 1. The the correlation kernel of the 
process V vanishes if at least one of the arguments is in X out , and on X- in x £ in 
it is equal to the normalized (k + l)th Christoffel-Darboux kernel for the Jacobi 
polynomials on (— |, |) with the weight function (| — x) z ~ k { \ + x) w ~ l . 

Proof. One way to prove this statement is to substitute integral z and w into the 
formulas of Theorem 10.1. A simpler way, however, is to use the asymptotic relation 



M n b n 



M(l + s) 



,m) =p^(«) + o(m- 1 ), 



M -> oo, 



where P^ 01 '^ is the nth Jacobi polynomial with parameters (ct,/3), see, e.g., [NSU, 
(2.6.3)]. The estimate is uniform in s belonging to any compact set inside (—1, 1). 
It is not hard to see that the weight function h(y) as well as the constants H m -i, 
A m -i, A m , see above, converge to the weight function and the corresponding con- 
stants for the Jacobi polynomials. Then Theorem 9.2 and Proposition 11.3 imply 
the claim. □ 



Appendix 

The hypergeometric series 3F2 evaluated at the unity viewed as a function of 
parameters has a large number of two and three-term relations. A lot of them were 
discovered by J. Thomae back in 1879. In 1923, F. J. W. Whipple introduced a 
notation which provided a clue to the numerous formulas obtained by Thomae. An 
excellent exposition of Whipple's work was given by W. N. Bailey in [Ba, Chapter 
3]. We will be using the notation of [Ba] below. 

Proof of (7.6). The formula (7.6) coincides with the relation 

Fp(0;4,5) = Fp(0;l,5), 

see [Ba, 3.5, 3.6]. 

Proof of Lemma 8.2. The right-hand side of (8.6) contains two 3F2's. We will 
use appropriate transformation formulas to rewrite both of them. 

For the first one we employ the relation 



Sin ^ 4 Fp(0) = ^ ^ , (A.l) 

7rr(a i 4 ) ' r(a 2 34)r(a 2 45)r(a345) r(ai23)r(ai2 5 )r(a!i35) 



which is [Ba 3.7(1)] with the indices 1 and 2 interchanged, and 3 and 4 interchanged. 
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By [Ba, 3.5, 3.6], we have 
Fp(0) = Fp(0;4,5) = 



1 



r(ai 23 )r(/34o)r(/35o) 



3-^2 



«145) «245, Ct345; 
AlO, ^50 



1 



Fn(l) = Fn(l;2,4) = 
1 



r(*)r(e)r(/) 
1 



3-^2 



a, 6, c; 
e,/ 



r(ai2 4 )r(/3 12 )r(/3 14 ) 

3-^2 



3-^2 



r(e - c)r(l + a - 6)r(l -b-c + f) 

1 



Q!l35) «013 5 «015! 
/5l2; A 4 

/ — b, 1 — 6, 1 — e + a; 
l + a-6, 1 - b - c + f 



Fn(4) = Fp(4;l,2) 
1 



r(ai24)r(/?4i)r(/?4 2 ) 
3-F2 



3-^2 



r(e - c)r(l + b + c - f)T(l + a + c-f) 
where s = e + f — a — b — a Thus, (A.l) takes the form 



«034, «045, «345; 
041, 

1 - / + C, 1 - s, c; 
1 + 6 + c - /, 1 + a + c - / 



3-^2 



+r 



a, 6, c; 
e,/ 

l-/ + a, s, e, /, b + c- f 
e — a, 6, c, e — c, 1 + a — 6 

1- f + a, e, f, -b-c + f 
f-c,f-b,e-c, 1 + a + c- f 



3F2 
3F2 



f - b, 1 - b, 1 - e + a; 
l + a-6, 1 - 6 - c + / 

1 - / + c, 1 - s, c; 
1 + b + c-f, 1 + a + c- f 



where we used the identity 

TV 



sin7r(l — b — c + f) 
Now set 



r(i-6- c +/)r(6+ c -/) = -r(-6- c +/)r(i+6+c-/). 



a = N , b = —z — w' , c = —z — w, e = u — z + 



N + 1 



-C 11 

J = —z — z — w — w , 



s = e + / — a — 6 — c = -u — z' 



iV-1 



Also, recall the notation £ = z + z / + «; + «/. Multiplying the last relation by 

r 



u + ^ 7 U - Z -M=k 
u-^,u-z + ^ 



we get 



U + ^5-=-, U — Z 



N-l 



N+l 



= r 



2 ' 

N+l 

2 ' 



3-^2 



u + ^P, 1 + A + E, u - z' - *f±, — E, -z + z' 



A, — 2; — w , —z — to; 

jV+l 

2 ' 

/ JV-l 



u- Z+ ^i, — E 



it — -^2^-, — u/, — w, u + iu + ^5-^, 1 + N + z + w' 



(A.2) 



X3^2 



V - W, Z + W' + 1, -It + Z + A±i; 



1 + A + z + w', 1 + z- z' 

+ { a similar expression with z and z interchanged }. 
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This is the transformation for the first term in (8.6). 
As for the second term, we use [Ba, 3.2(2)] which reads 



3-^2 



a, 6, c; 
e,f 



= r 



1 - a, e, /, c- b 
b, f -b, 1 + b - a, c 



3-^2 



6, 6-e + l, 6-/ + 1; 
1 + 6- c, 1 + b - a 



+ { a similar expression with b and c interchanged }. 



Set 



a = —N + 1, b = z + w' + 1, c = z' + it/ + 1, e = E + 2, / = u + «/ 



iV — 3 



We get 



3-^2 



= r 



-iV + 1, z + w' + 1, z' + w' + 1; 
E + 2, u + u/- ^ 

AT, E + 2, u + - 



jV-3 / _ 
. z z 



X3-F2 



z' + w + l, u-z - + 1 + z + w', z' + + 1 

jz + «;' + 1, -z' - w, -u + z+ ^ 1 
1 + z-z', 1 + N + z + w' 

+ { a similar expression with z and z' interchanged }. 

Let us multiply this by the prefactor of the hypergeometric function in the second 
term of (8.6). Recalling the formula for h^-i = h(N — 1, z, z', w, w'), see (7.2), and 
canceling some gamma-factors we obtain that the second term of (8.6) equals 

u + -u + 1 + N + E, z + w + 1, z + w' + 1, z' - z 



2 , - , 2 

-u + z' + u + io + Z + l,l + N + z + w' 



X ■ 



sin7r(w — z — 



JV-l- 



7T 



F(u) 3F2 



2; + w' + 1, — — w, —u + z + 
1 + z-z', 1 + N + z + w' 



N+l . 
2 ' 



(A.3) 



+ { a similar expression with z and z' interchanged }, 
where we also used the identity 



N-l 



)r(-u + z+*P) = - 



7T 



sin n(u 



N-l 



) 



(Recall that F(u) was defined right before Lemma 8.2.) 

Now, in order to get (8.6) we have to add (A. 2) and (A.3). Since both expression 
have two parts with the second parts different from the first parts by switching z 
and z\ it suffices to transform the sum of the first parts. We immediately see 
that the hypergeometric functions entering the first parts of (A. 2) and (A.3) are 
identical. By factoring out the 3^2 's and some of the gamma-factors, and using 
the identity r(r)r(l — r) = 7r/sin7rr several times, we obtain that the sum of the 
first parts of (A. 2) and (A.3) equals 



1 + iV + E, z + w + 1, z + w' + 1, z' - z 



-u + z' + u + w + ^±1, E + l, 1 + N + z + w' 



2 

X 3^2 



z + w' + l, 

1 + Z 



-u + z + 
1 + N + z + w' 



N+l. 
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multiplied by 



sin 7r(u — 



N-l 



smn(z + w') sin7r(^ + w) 



sin7r(tt — z' — -^y^-) sin7rS 



+ sin7r(-u-z- F{u) 



(A.4) 

Observe that F(u) as a function in u is a linear combination of 1/ sin7r(— u + 
z + ^y^) and l/sin7r(— u + z' + ^-^)- Thus, (A.4) is a meromorphic function. It 
is easily verified that all the singularities of (A.4) are removable and (A.4) is an 
entire function. Moreover, since the ratios sin(tt + a)/ sin(-u + (3) are periodic with 
period 2tt and are bounded as — > ±00 (for arbitrary a, j3 e C), F{u) is bounded 
on the entire complex plane. By Liouville's theorem, F(u) does not depend on u. 
Substituting u = we see that (A.4) is equal to — sinnz/n. 

This immediately implies that the sum of (A. 2) and (A. 3) is equal to the right- 
hand side of (8.4), and the first part of Lemma 8.2 is proved. 

Now let us look at the formula (8.7). Note that the hypergeometric functions in 
(8.7) can be obtained from those in (8.6) by the following shift: 



N i-> N + 1, z^ z 



1 

2 ; 



z — 



w 1— > w — 



w' 1— > w' — 



We use for them exactly the same transformation formulas as we used for (8.6). 
By computations very similar to the above, we obtain that the first term of (8.7) 
is equal to 



-r 



u + -u + ^±i, N + 1, E, z' - z 



2 , - , 2 

-u + z' + ^f 1 , U + W + ^2^, N + l + z + w', z' + w, z' + w' 
^+r^) smn(z + w') s\mi{z + w) 



x • 



sin7r(w 



X 3 ^2 



7rsin7r(w — z' — ^-2^-) sin7rS 
-z' -w + 1, z + w', -u + z + 



(A.5) 



N + l + z + w', 1 + z-z' 
— {a similar expression with z and z interchanged }, 

while the second term of (8.7) is equal to 

z' + w, z' + w 



u + 



-u + z' + 



' 1 N ~ 1 ~ + w+Z±±, N+l + z + w'. ~' ' -• 7 ! 



u 



X ■ 



sin7r( , u — z 



N-l 



7V 



F(u) 3 F 2 



-z' - w + 1, z + w', -u + z + ^2^: 
N + l + z + w' 1 + z-z' 



(A.6) 



+ { a similar expression with z and z' interchanged }, 



Adding (A.5) and (A.6) and using the fact that (A.4) is equal to — smnz/iv, we 
arrive at the right-hand side of (8.5). □ 

Proof of Lemma 8.4. We start with deriving a convenient transformation formula 
for 3F2. [Ba 3.7(6)] with indices 4 and 5 interchanged reads 



sin 7r/?4o-Fp(0) 



7rr(a 45)r(ao34)r(ao24)r(aoi4) 
Fn(0) 

r(a345)r(a245)r(ai45)r(ai34)r(a 2 34)r(ai24) 
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(A.7) 



+ K Fn(4), 



where 



sin 7rai45 sin 7rct245 sin ^0:345 + sin 7r 01123 sin tt04q sin 7r/35o 



7T- 



By [Ba, 3.6] we have 



Fp(0) = Fp(0;4,5) 



r(ai 23 )r(/34o)r(/35o; 



3-^2 



1 



Fn(0) = Fn(0;4,5) = 



r(a)r(e)r(/) 
1 



3-^2 



a, 6, c; 
e,f 



r(a 45)r(/3o4)r(/3o5; 



3-^2 



«145, «245, CK345; 
040, 050 



«023 5 «013, CK012; 
A)4, A)5 



r(i- a )r(2-e)r(2-/) 

1 



3-^2 



1 — a, 1 — 6, 1 — c; 
2-e, 2-/ 



Fn(4) = Fn(4;0,3) 



r(ao34)r(/3 40 )r(/343) 



3-^2 



3-^2 



r(i-/ + c)r( e )r(i + a + 6-/) 

sin ixa sin 7r6 sin tic + sin 7rs sin Tie sin 7r/ 



«124, «145, Ct245! 
^40 , Al3 

e — c, a, b; 
e, 1 + a + b-f 



K = 



TV'' 



where s = e + f — a — 6 — c. Then (A. 7) takes the form 



3-^2 



7T 



sin 7re 
sin7re 



a, 6, c; 
e, f 

l-/ + c, 1-/ + 6, l-/ + a, s, e, / 
c, 6, a, e — 6, e — a, e — c, 2 — e, 2 — / 

1-8, 1-f + b, 1-f + a, s, f 
1+a+b-f 



3 r 2 



3-^2 



1 — a, 1 — 6, 1 — c; 
2-e, 2-/ 

e — c, a, 6; 
e, 1 + a + b- f 



(A.8) 

with iiTo as above. 

Let us apply (A.8) to both hypergeometric functions in (8.6). For the first one 
we set 



a = N, b = —z — w' , c = —z — w, e = — E, / = u — z + 



N +1 



s = e + f — a — b — c = u — z' 



N — 1 



Then K = — sin ttT, sin7r(it — z' — ^o-^-) sin7r(it — z + ^+^-)/tv 3 , 11 and rewriting the 



11 Here we used that N € Z and sin7ra = sinnN = 0. 
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sines in K as products of gamma-functions, we have 



3-^2 



N, —z — w', —z — w; 



u — z + 



N+l 



= r 



7V-1 / N—l i 

-u — w — — u — w — £ +- L , — u + z 



N+l 



u — z — 



l N-l 



— E, u — z + 



JV+1 



2 ' 2 ' ' ' 2 ' 2 ' ' 1 2 

-z -w, -z- w', N, -z' — w, — E — AT, -z' —w', 2 + E, —u + z — ^ 



n 



x 



sin7rE 



iF 2 



-N + 1, z + w' + 1, z + w + 1; 
E + 2, -u + z- ^ 



+T 



-u-w'- V' 



-u + 2 + 
-U + -2 — 



JV+1 
2 

AT-1 



3-^2 



—2' — to', AT, — z — w'; 
-E, -u-w'+^ 



For the second one we set 
a = -N + l, b= z + w' + 1, c = z' + «;' + !, e = E + 2, f = u + w' 



(A.9) 
N-3 



s = e + / — a — 6 — c = m + to + 



Then K = sin7rE smn(u + w + ^4^-) sin7r(tt + w' 



N-3 



)//T 3 . Observe that the 



first term in the right-hand side of (A. 8) will now vanish thanks to the T(a) in 
the denominator (remember that N G {1,2,...}). Rewriting sines as products of 
gamma-functions again, we get 



]F 2 



-u + z + 



N+l 



-u-w' + ^.-u + z-^ 



-N + 1, z + w' + 1, z' + w' + 1; 
E + 2, u + w' - ^ 

A ~ l 1 v z + w + l, -N + l, z + w' + 1; 



-u — w 



3 r 2 



E + 2, -u + z - 



N-3 



(A.10) 

Now let us substitute (A.9) and (A. 10) into (8.6). Observe that the two siVs 
from the right-hand sides of (A.9) and (A. 10) are obtained from the 3-F 2 's from 
(8.6) by the change 



u I— > —u, (z, z ', w, w') i— > (w' , w, z', z). 



(A.ll) 



Our goal is to show that the prefactors of these siVs after substitution will be 
symmetric to the prefactors of 3F 2 's in (8.6) with respect to (A.ll). 



One prefactor is easy to handle. The coefficient of 3-F2 
after the substitution of (A.9) and (A. 10) into (8.6) equa^ 



-z' - w', N, 
-E, -u-w' + 



-z — w 1 ; 

N+l 
2 



u — ^+P-, u — z + 



= r 



N-l 
2 

JV+1 
2 

-u + 



-u — w 



N-l 



—u + w' — 



2 ' 
AT-1 " 



-u + z + 
-u + z — 



N+l 
2 

N-l 



—U — 



N+l 

JvV - U + W' + ^ 



which is symmetric to V 



u+ N±l^ u _ z _N fl 
u ^N f l^ u _ z+ N+l 
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with respect to (A.ll). 



As for the other prefactor, the verification is more involved. Namely, we need to 
prove the following equality: 



— u — 



N-l 



i N — l i , N+l i N — l i 

W — ^rr^, —u — w — ^^rr*-, —u + z + £ ^- L , u — z — il rr--, — E, u — z + 



N+l 



xF 



2 ' 

w, —z 



w', N, - : 



2 ' 
/ 



UK 



-E — N, 



2 ' 

w', 2 + E, -u + z- ^ 



xF 



u+ N±l^ u _ z _N f l 

u-^,u-z + ^ 



i 
TV 



sin7rE 



+r 



-u + z + ^, -u + z' + H±±, u + w+^,u + w' 



-u + z + -u - w' 



N-l 



X 



N-3 
2 

1 



sin7r(z + w) sin7r(^ + w') sinyrz' sin7r(z / + w) sin7r(z / + to') sin7rz 



h(N — 1, z, z', w, w') sin 7rE sin tt(z' — z) 



sinit(-u + z ! + 



sin7r(-w + z+ 



- u+ E±1, u+ n±i 

u + w > + N±1 jU + w+ E±1 j - U + z > + N±1^ U + Z -N^1 



X 



X ■ 



h(N — l,w' ,w, z' , z) sin7rE sin7r(«; — w') 
sin tt(z' + w') sin n(z + w') sin tiw sin tt(z' + w) sin ir(z + w) sin nw' 



sin7r(w + w + 



sirin(u + w' + ^±±) 



After massive cancellations 12 this equality is reduced to the following trigonometric 
identity (here y = u — ^-^): 



sin7n/ sin7r(z + w) sin7r(V + w) sin7r(z + w') sin7r(z / + w') 
sin7r(?/ — z) sin7r(y — z') smn{y + w) sin7r(y + w') 
sin 7r(z' + w) sin 7r(z' + w') sin ttz sin tt(z + w) sin tt(z + w') sin tvz' 
sin7r(z — z') sin7r(y — z) ' snm{z' — z)sirm{y — z') 
sin n(z + w') sin tt(z' + w') sin tiw sin 7v(z + w) sin tt(z' + w) sin nw' 



sin n(w — w') sin n(y + w) 



+ 



smn(w' — w) sin7r(y + w') 



One way to prove this identity is to view both sides as meromorphic functions 
in y. Then it is easily verified that the difference of the left-hand side and the 
right-hand side is an entire function. Moreover, both sides are periodic with period 
2/T and bounded for \Qy\ large enough. This implies that both sides are identically 
equal. The proof of Lemma 8.4 is complete. 

On the proof of Lemma 8.5. This proof is very similar to that of Lemma 8.4 
above. The needed transformation formulas for the siVs are obtained from (A. 9) 
and (A. 10) by the shift (N, z, z', w,w') i-> (N + 1, z - \, z' - §, w - \, w' - \). 
After substituting the resulting expressions into (8.7) we collect the coefficients of 
3-F 2 's and compare them with what we want. As in the proof of Lemma 8.4, one of 
the desired equalities follows immediately, while the other is reduced to the same 
trigonometric identity. 



12 which also rely on the fact that N 6 Z, cf. the previous footnote. 
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On analytic continuation of the series 3^2(1). Here we prove that the function 



T(e)T(f)T(e + f-a-b-c) 



a, 6, c 
e, f 



can be analytically continued to an entire function in 5 complex variables a, 5, c, e, /. 
We stated this claim in the beginning of §7 and used it in §8. 
Apply the transformation formula 



3-^2 



a, b, c 



f 



1 



e, /, s 
a, s + 6, s + c 



3-^2 



e — a, / — a, s 
s + 6, s + c 



1 



where s = e + f — a — b — c. It allows us to conclude that the function in question 
continues to the domain dt(a) > (other parameters being arbitrary). 

Likewise, we can continue to the domain 3?(6) > and also to the domain dt(c) > 
0. Then one can apply a general theorem about 'forced' analytic continuation of 
holomorphic functions on tube domains: see, e.g., [H, Theorem 2.5.10]. 
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